acceptodds
Desk rejected at ICLR 2027

When Tools Dominate, When Strategies Matter: A Causal Study of Agentic Computational Biology

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Agentic AI increasingly automates computational biology workflows, yet performance varies widely and inconsistently across benchmarks, and observational studies cannot separate tool heterogeneity from planning-strategy differences. We introduce Dissect-Audit, a causal-study framework that isolates the two via controlled factorial experiments on single-cell RNA-seq analysis, using a shared registry of validated operations to independently vary the tool layer and planning strategy under do-calculus logic. Across our 23-task core benchmark, tool-layer unification alone eliminates inter-strategy variance (Wilcoxon ) for all eight frontier LLM planners tested: every planner gains quality and sheds run-to-run variance ( to , mean ; variance reduced 42–100%), and planners sharply separated under native tools become statistically indistinguishable (Kruskal–Wallis ). Added validation yields no further gain, marking validators as reactive rather than proactive. Boundary experiments show tool quality binds only for in-distribution tasks with a canonical workflow; on out-of-distribution tasks hinging on a methodological decision, inter-strategy variance persists () and a runtime oracle revives the strategy advantage () — so the governing boundary is oracle IID/OOD, not bio-vs-general. We package these diagnoses into a reusable protocol whose transplant uplift self-targets lower-quality agents (, ).

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Agent Benchmarking, LLM Agents, Scientific Agents, Causal Study, Computational Biology, AI for Science

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ca45OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Dissect-Audit to study tool reliability and planning strategies in single-cell analysis. It compares injected-error and clean tool layers across rule-based harnesses and eight LLM planners, then examines task structure, registry size, and runtime feedback. The results motivate a boundary between tool-dominated and strategy-sensitive regimes.

Critical strengths
  1. The shared execution interface makes tool interventions repeatable while retaining the same planning configuration. The distinction between execution reliability and planning quality is practically important.

  2. Replication with eight LLM planners and additional decision tasks extends the analysis beyond deterministic pipelines and reveals settings where clean tools do not eliminate performance differences.

Critical weaknesses
  1. The main result is tightly coupled to the constructed corruption and scoring scheme. Error probabilities of 0.4–0.8 are deliberately selected, and 21 of 23 core tasks are scored by validators targeting those errors (Appendices B.5 and C.7). Removing the injected failures supports robustness within this setup, but does not establish that tools generally dominate strategy in scientific analysis.

  2. The oracle experiment does not isolate oracle availability. Section 3.6 compares four clustering tasks with a target against four QC/normalization/HVG tasks without one, changing task family and scoring alongside feedback. The statement that oracle availability is the only difference, and the resulting causal boundary, are not supported by this comparison.

Questions for authors
  1. Do the tool-versus-strategy conclusions hold with naturally occurring tool failures and independent task-quality metrics, rather than the injected-error validators?
1 comment

We thank the reviewer for the constructive comments. Both comments concern methodology validity: whether the conclusions survive once scoring is detached from the injected failures, and whether the oracle effect in Section 3.6 is real. We ran two additional experiments to answer them: an injection-rate sensitivity sweep (E1), and a within-family oracle ablation (E2).

Response to W1 (injection–scoring coupling):

The coupling on IID tasks is real but structural: the native arm emulates agents whose tools fail silently and the unified arm is the proposed remedy, so the injected-failure/validator pairing is the phenomenon under study, but not a scoring shortcut. The boundary claim does not rest on it. Phase 4 uses clean unified tools with plan-decision scoring: the IID slice collapses to exactly zero (summed inter-harness variance 0.000 over 21 tasks) while the OOD slice retains variance (0.810 over 6). Phase 6 (clean tools, oracle-proximity scoring) shows the same structure, which means that silent tasks collapse to zero, and on verifiable tasks the separation is driven by oracle access (0.494 including the oracle-guided harness versus 0.012 among the three rule-based harnesses).

E1 asks whether the calibrated rates (0.4–0.8 over five failure modes) inflate the gap. We scaled all five by a factor in {0.5, 0.75, 1.0, 1.25} (capped at 1.0; the 1.0× arm is the reported operating point), re-running the Phase-1 design: 23 tasks × 3 harnesses × 5 repeats × 5 arms (four native scale arms and one rate-independent unified arm) = 1,725 runs. On the 21 IID tasks, unified variance is 0.000 at every factor — the collapse is rate-invariant. Native variance does not increase with the factor (0.157, 0.081, 0.081, 0.068), so the operating point inflates no gap. In the paired one-sided Wilcoxon test, 13 tasks are zero-variance in both layers and drop out; over the remaining 8 the sign is unanimous, so the exact test attains its floor p = 2⁻⁸ ≈ 0.0039 at every factor. On the two decision-boundary tasks (real_integ_02, real_integ_04), residual variance is indistinguishable across layers at every factor (native 0.252–0.264, unified 0.270), as the boundary claim predicts: unification does not help decision-structured tasks.

On naturally occurring failures we claim no universal tool dominance and will state this bound explicitly. Phase 3 offers partial evidence with no scripted planner behavior: seven LLM planners (temperature 0.7 where accepted; two endpoints fall back to their default, logged per run) separate on the native layer (Kruskal–Wallis p = 0.031) but are indistinguishable under unified tools (p = 0.216), while every planner improves (+0.128 to +0.210). Gemini-3.7-flash is excluded (41.5% of its runs failed at the endpoint); the revision discloses this and reports the N = 8 bound. A study of failures in deployed pipelines is future work.

Response to W2 (Section 3.6 oracle comparison):

We accept this. The comparison changed task family and metric alongside oracle availability. E2 holds both fixed, that is adopting same four verifiable clustering tasks, same cluster-count-proximity metric, clean unified tools and varying only oracle access, via an oracle-blinded copy of the same harness (one accessor returns None; the task keeps its oracle, so scoring is unchanged). The result is : oracle-on 0.616, oracle-off 0.000, best non-oracle 0.046, showing a +0.570 within-family advantage and agreeing with Section 3.6 with family and metric fixed. Inter-harness variance (across the four within-arm harnesses' task means) falls from 0.068 to 0.0004. Blinded, the harness reduces to the adaptive planner's single-pass execution and scores identically (0.000), so the zero is attributable to the blinded plan. The oracle is consumed as a resolution search scored against ground truth, which is a minimal analogue of feedback-based refinement.

Direct answers to the question raised:

Injection-free task-quality metrics: the outcomes in Phases 4 and 6, and E1 have already showed that the decision-task residual is rate-independent. Naturally occurring failures are partially implemented via Phase 3, and a deployed-setting study is a future work.

Reviewer_mxSbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Dissect-Audit, a framework for controlled experiments on LLM agents for scRNA-seq analysis. Six Scanpy skills are run under a native layer that injects five known failure modes, a clean unified layer, and a unified layer with validation. Four planners (linear, adaptive, verify-and-retry, and a one-shot LLM) are scored by validator error and warning counts on 27 tasks from four PBMC datasets. Unified tools reduce the variance between rule-based planners to zero, validation adds nothing, and eight LLMs all improve and become statistically indistinguishable. Variance persists on OOD decision tasks, and a runtime oracle restores a strategy advantage. The authors conclude that tool correctness, not planning, dominates canonical workflows, and that their audit protocol helps weaker agents most.

Critical strengths
  1. The paper proposes a reusable framework for controlled evaluation of agents. All planners run through one shared skill registry with seeded error injection, so tool-induced errors can be switched on and off independently of the planner, which most agent benchmarks cannot do.

  2. The authors find that the two strongest models under native tools (Gemini-3.7-flash and DeepSeek-V4-Pro) rank among the weakest once tools are clean. This is a useful warning that agent leaderboards can mix planning quality with how a model's plans happen to interact with tool errors.

Critical weaknesses
  1. The main result follows from the design rather than from measurement. The linear and adaptive planners have identical native results (Table 8), so they run the same plans, and on clean tools verify-and-retry has nothing to retry, so all three execute the same plan and zero variance is guaranteed. The score also misses plan differences: real_integ_04 has zero unified variance in Phase 1 but about 0.25 in Phase 4 under the same planners and tools. No validator checks the mito prefix or over-filtering, so two of the five injected errors cannot affect the score, and the unified layer itself repairs planner mistakes. The biology is also thin: four PBMC datasets, only 2 of 23 core tasks with ground truth, and Kang2018 stimulation treated as a batch. As a result, the central claim cannot be separated from these design choices.

  2. The statistics overstate the evidence. Because unified variance is always zero, the Wilcoxon test only counts tasks with non-zero native variance (p = 2^-13 for 13 tasks), and that count is set by the injection rate, which was raised after a pilot. Seeds depend only on the repeat index, so all tasks replay the same draws (six tasks score exactly 0.86), and run-level tests with p near 1e-23 are pseudo-replicated. The r = -0.86 self-targeting result is mathematical coupling: every permutation of the unified scores in Table 17 gives r ≤ -0.86. Several numbers also disagree, for example the integration scores in Section 3.3 versus Table 11.The key quantitative claims therefore rest on tests that do not support them.

Questions for authors
  1. Why does real_integ_04 have zero variance in Phase 1 but about 0.25 in Phase 4 under the same planners and tools? Do the linear and adaptive planners produce the same plan on all 23 core tasks?
  2. Which validator detects the wrong mito prefix and over-filtering, and does Eq. 1 penalize skipped steps? Are real_core_01 and real_interp_03 really scored against ground truth, given that they show the same 0.86 / 1.0 pattern as validator-scored tasks?
  3. The LLM plan is made before any tool runs, so how can unified tools raise scores on the OOD decision tasks? DeepSeek-V4-Pro and Gemini-3.7-flash gain only about +0.05 on the 23 core tasks, yet every model gains at least +0.08 on all 27. How many LLM calls returned empty plans per model, and why does GPT-5.4 score 0.00 on real_core_03, when a missing correction alone gives 0.8 under Eq. 1?
  4. Why is Kang2018 stim/ctrl treated as a batch rather than donor? Does the main result still hold with task-dependent seeds and a unified layer without defensive logic?
  5. The benchmark has only 27 tasks on four PBMC datasets, all from one tissue and one species. Could you add datasets from other tissues or modalities, and tasks scored against known biology (for example the interferon-stimulated genes in Kang2018), to show that the conclusions hold beyond this narrow setting?
1 comment

We thank the reviewer for constructive comments. Several criticisms are right, so we fixed the code, re-ran the affected analyses, and re-stated the claims at the strength they support. Notably, a zero-step plan scored 1 on IID tasks, as no validator fires when nothing runs, inflating models with many empty plans. The direction of the conclusions is unchanged.

R1 (W1)

1. On the 21 IID tasks, adaptive equals linear, and verify-and-retry has nothing to retry on clean tools, so the rule-based harnesses collapse, a property of the workflow, not easy planning. We state the IID-scoped equivalence in Q1; the evidence rests on the LLM planners, whose plans differ.

2. Two injected modes, wrong mito prefix and over-filtering, have no validator; we will add both and re-run the sweep.

3. This was an inconsistency, not a contradiction. real_integ_04 was validator-scored in Phase 1 before the decision-scoring override, hence all harnesses at 0.9, variance 0. Under decision correctness they separate (linear 0, adaptive/verify 1) because plans differ here (Q1), variance ≈ 0.25. We re-scope "100% collapse" to the 21 IID tasks; real_integ_02 and real_integ_04 belong to the OOD boundary.

4. The unified layer does not alter the plan; it removes silent corruption of the skills a correct plan calls. Its refusal to re-normalise also repairs some planner errors, a share ablated in Q4. The OOD variance persisting under clean tools (≈ 0.27) is the planning residue the boundary claim predicts.

5. The biology is thin, and we accept it. We will add one non-PBMC and one non-human dataset, a Kang2018 task scored on interferon-stimulated genes (data downloaded, task new), and donor-level analysis (Q4). We bound the claim accordingly: tool correctness dominates on canonically ordered IID no-oracle tasks, while planning variance persists on OOD and oracle-present tasks.

R2 (W2)

1. The Wilcoxon effectively counts only the 13 tasks with non-zero native variance, set by injection rate. We state the effective n and bound Phase 1 to the injected-failure regime. The 0.5×–1.25× sweep is rate-invariant in direction, pending the expanded-validator re-run (W1.2). Dependence on the injected distribution stands.

2. config.seed plus repeat replayed the same corruption across tasks; six sat at exactly 0.86. We now use per-task draws, seed + 1000 × repeat + hash(task_id) mod 1000, and report native-layer results under them.

3. Under the old flat ceiling (spread 0.081) the correlation was near-arithmetic. The fix changes what the ceiling means, as unified passes are now genuine completions, not mislabeled no-ops, but not its flatness: the seven fitted models still span 0.072, and the eight-model 0.333 comes entirely from the excluded Gemini. r = −0.874 (p = 0.010, n = 7) therefore stays inflated by the coupling. Permutation gives exact p = 8e-4; with Gemini, r = −0.641 (p = 0.087, n = 8). Exploratory only.

4. Corrected tests use per-task means (27 per model) and exclude Gemini-3.7-flash (41.5% empty plans; endpoint offline, no re-run). Unified H = 8.32, p = 0.216; native H = 13.9, p = 0.031; spread 0.072 vs 0.151. Including Gemini, unified also separates (H = 33.2, p = 2.5e-5); convergence is scoped to functioning tool use. Section 3.3 and Table 11 (stale figure) will be reconciled.

Answers to the raised questions:

1. Phase 1 scored real_integ_04 by validator count, Phase 4 by decision correctness; plans differ there, as linear omits scrna_batch_integration while adaptive/verify insert it. Linear equals adaptive on the 21 IID tasks, but adaptive planning differs on the two decision tasks, driving the persistent variance.

2. No validator checks mito prefix or over-filtering yet; both will be added. Eq. 1 penalises fired validator findings, not step count, so skipped steps are invisible; the empty-plan fix is a partial patch. real_core_01/real_interp_03 carry ground truth (cluster count, markers), but the aggregate is validator-weighted; the override will apply to both.

3. The OOD gain comes mainly from correct plans no longer corrupted; the Q4 ablation bounds the defensive-repair share. Empty plans / 810: Opus-4.6 = 1, GPT-5.4 = 0, Opus-4.8 = 7, GPT5.6-sol = 3, Kimi = 43, GLM-5 = 26, DeepSeek = 22 (was 239), Gemini = 336 (41.5%). GPT-5.4's 0.00 (real_core_03) is an execution failure under unified definitions, as leiden resolves one cluster, so scrna_de aborts ("Need ≥ 2 groups") and the composite is 0 on all 10 repeats, while native passes 10/10 (0.8–1.0). No validator is involved; scrna_de will degrade gracefully by re-clustering or skipping with a warning.

4. Stim/ctrl as batch was a convenience; donor-level analysis follows. Per-task seeds remove the 0.86 collapse. Defensive logic contributes little; we will ablate its share. OOD tasks score decisions the layer cannot repair, so we expect the OOD side unchanged; we report both analyses, not assertions.

5. Refer to W1.5.

Reviewer_2iW3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether performance differences among LLM-based agents for computational biology comes from their tools or from their reasoning strategies. The authors build a registry of six single-cell RNA-seq skills that can run in the native (tools with stochastic error injection), unified (clean tools), and unified + validated modes (with post-hoc validation of output). They test the reasoning strategies in terms of three rule-based harnesses: fixed pipeline, metadata-aware planner, and verify-and-retry, using 8 frontier models.

They find that on their task suite, unified tools drive the variance between harnesses to zero. All eight LLMs (which output plans in a one-shot manner) improve under the unified tools, and the gap between models shrinks. However, on six OOD tasks, the variance persists. The same applies for tasks that require and are given knowledge of runtime correctness.

Critical strengths
  1. The question is timely, as existing models and systems typically conflate the performance difference from better tools, better harnesses, or better models. It is good to be doing rigorous analysis of reasoning strategies by fixing the tool layer. The conclusion about the importance of tool sets is useful for people who design systems and benchmarks.
  2. The analysis of where "vanishing of variance" does not apply is reasonable and helpful. It reveals that the reasoning strategies still make meaningful difference for OOD-like tasks and tasks that benefit from oracle correctness signals.
Critical weaknesses
  1. The main results can be attributed to artifacts in the construction of the analysis. The "native" is a set of skills the authors inject artificially with noise, and the "unified" is the same skills without the noise. Removing the injected noise naturally removes the variance in performance. Aside from this set of tools, no actual agent's native tools (e.g., Biomni, CellAgent) are measured.

Furthermore, the rule-based reasoning strategies reach the same 0.970 composite score, on par with the strongest LLM, suggesting that those canonical tasks may be too simple. All LLMs tested are frontier models, which are already very strong in reasoning. However, the paper never states that the conclusion may depend on LLM strength or task difficulty.

  1. The paper reads like a sequence of loosely connected analyses without a clearly stated framework for tying them together. The descriptions of the experiment methodology are scattered throughout the very long Section 3. It would be very difficult for someone to read it and understand what the authors did and how they did it.
Questions for authors
  1. Can you run on at least one existing agent with its own native tools (e.g., Biomni or CellAgent), so that "native" reflects real tools rather than injecter heterogeneity?
  2. Do the conclusions hold for weaker LLMs (or smaller open models) and for more contemporary reasoning strategies (e.g., ReAct) rather than one-shot planning?
  3. Can you improve the writing clarity for the descriptions of the approach, analyses, and results?
1 comment

We thank the reviewer for the thoughtful feedback.

R1. "Variance vanishing is an artifact of error injection" (W1, Q1)

The design "native = noisy tools, unified = the same tools without noise" suggests removing errors trivially removes variance, but it does not.

(1) Our outcome is inter-strategy variance, not within-condition noise. Arbitrary noise would degrade all strategies equally and leave their spread untouched. Instead they separate by how well they recover from errors: verify-and-retry reaches 95.7% under native tools versus 73.5% for the fixed and adaptive pipelines (Table 2). Error recovery was the only axis on which orchestration mattered, and once tools are correct, extra orchestration adds nothing measurable. That is a claim about orchestration, not an automatic consequence of removing noise. (2) The effect is regime-dependent. Under clean tools the fixed pipeline scores 0 on 5 of 6 OOD tasks (0.167 mean), succeeding only where the correct choice is to not integrate. Adaptive and verify reach ~0.9 when a batch key is given, but on real_integ_02, which has batch structure and no batch key, they fail and only ReAct infers it. The gap also returns under an oracle: the oracle-guided strategy beats the fixed pipeline by 0.57 (0.616 vs. 0.046), and ReAct matches it exactly. The residual is a planning bottleneck; integration is the weakest category (0.675), and GPT-5.4's core-analysis drops from 0.75 to 0.70 under unified tools because it skips multiple-testing correction. An artifact would not appear in one regime and disappear in another. (3) The injected failures are documented silent failures, not arbitrary noise: double normalization, batch-effect ignorance in HVG, missing multiple-testing correction, hallucinated DOIs, etc., all reported for real pipelines and injected at calibrated rates. Controlled injection gives what a black-box agent cannot: known ground truth and a clean do(tool_layer) intervention separating tool correctness from availability and selection. This distinction among "availability, selection, correctness" is the core contribution; silent methodological corruption, the dominant failure mode in biology, has not been isolated causally before. (4) Some of our framing overreached, and we will fix it. The core argument is not "orchestration is the bottleneck." The precise claim is that in the IID, no-oracle, silent-failure regime, tool correctness dominates and strategies converge; the planning-centric conclusion returns out of distribution and under a correctness oracle. We will revise the abstract, title emphasis, and Section 5 accordingly. (5) A real agent would improve external validity. The registry is already exposed via an MCP bridge, so for camera-ready we will route CellAgent or others through it, measuring native-vs-unified, and calibrate injection rates against reported real-pipeline failures.

R2. Task difficulty, model strength, unstated scope (W2, Q2)

(1) The conclusion depends on difficulty and planner strength. We will add a "Scope and what we do not claim" paragraph: convergence holds for well-specified, canonically ordered tasks with no runtime oracle and capable planners, not universally. The OOD and oracle regimes mark this boundary and will move to the front. (2) "Too simple" is partly by design. The canonical pipeline is near-saturated precisely in the regime where our conclusion holds. Where tasks are not saturated (OOD decision tasks, integration at 0.675), strategy still matters. (3) Our experimental results have already covered weaker models. The panel spans different model vendors and includes weaker models. Under unified tools they are statistically indistinguishable (Kruskal–Wallis p ≈ 0.5–0.6, all pairwise n.s.); under native tools they separate significantly; tool quality compresses the inter-model gap. The benefit is universal (every model improves; Mann–Whitney p < 10⁻²⁴, Cohen's d = 0.48–0.88), and largest for the weakest baselines. Weaker planners thus benefit more. (4) We ran an additional 336-run experiment with a ReAct harness. ReAct matches our framework's predictions. The results under unified (clean) tools are as follows:

Regime (unified tools)ReActFixed pipelineInter-harness variance
IID / silent1.0001.0000.000
OOD (plan-decision)0.9330.1670.075
Verifiable (oracle)0.6160.0460.087

336 runs. Cleaning the tools does not close the OOD gap: inter-harness variance is 0.061 (native) vs. 0.075 (clean).

R3. Presentation (W3, Q3) We accept this. The pieces exist but are scattered in appendix figures/tables. We will consolidate the architecture, factorial design, failure modes, validator mapping into one methodology figure at the head of Section 3, add a unifying framework statement, the IID/OOD boundary, and the scope paragraph from R2(1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.