- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces Dissect-Audit to study tool reliability and planning strategies in single-cell analysis. It compares injected-error and clean tool layers across rule-based harnesses and eight LLM planners, then examines task structure, registry size, and runtime feedback. The results motivate a boundary between tool-dominated and strategy-sensitive regimes.
Critical strengths
-
The shared execution interface makes tool interventions repeatable while retaining the same planning configuration. The distinction between execution reliability and planning quality is practically important.
-
Replication with eight LLM planners and additional decision tasks extends the analysis beyond deterministic pipelines and reveals settings where clean tools do not eliminate performance differences.
Critical weaknesses
-
The main result is tightly coupled to the constructed corruption and scoring scheme. Error probabilities of 0.4–0.8 are deliberately selected, and 21 of 23 core tasks are scored by validators targeting those errors (Appendices B.5 and C.7). Removing the injected failures supports robustness within this setup, but does not establish that tools generally dominate strategy in scientific analysis.
-
The oracle experiment does not isolate oracle availability. Section 3.6 compares four clustering tasks with a target against four QC/normalization/HVG tasks without one, changing task family and scoring alongside feedback. The statement that oracle availability is the only difference, and the resulting causal boundary, are not supported by this comparison.
Questions for authors
- Do the tool-versus-strategy conclusions hold with naturally occurring tool failures and independent task-quality metrics, rather than the injected-error validators?
We thank the reviewer for the constructive comments. Both comments concern methodology validity: whether the conclusions survive once scoring is detached from the injected failures, and whether the oracle effect in Section 3.6 is real. We ran two additional experiments to answer them: an injection-rate sensitivity sweep (E1), and a within-family oracle ablation (E2).
Response to W1 (injection–scoring coupling):
The coupling on IID tasks is real but structural: the native arm emulates agents whose tools fail silently and the unified arm is the proposed remedy, so the injected-failure/validator pairing is the phenomenon under study, but not a scoring shortcut. The boundary claim does not rest on it. Phase 4 uses clean unified tools with plan-decision scoring: the IID slice collapses to exactly zero (summed inter-harness variance 0.000 over 21 tasks) while the OOD slice retains variance (0.810 over 6). Phase 6 (clean tools, oracle-proximity scoring) shows the same structure, which means that silent tasks collapse to zero, and on verifiable tasks the separation is driven by oracle access (0.494 including the oracle-guided harness versus 0.012 among the three rule-based harnesses).
E1 asks whether the calibrated rates (0.4–0.8 over five failure modes) inflate the gap. We scaled all five by a factor in {0.5, 0.75, 1.0, 1.25} (capped at 1.0; the 1.0× arm is the reported operating point), re-running the Phase-1 design: 23 tasks × 3 harnesses × 5 repeats × 5 arms (four native scale arms and one rate-independent unified arm) = 1,725 runs. On the 21 IID tasks, unified variance is 0.000 at every factor — the collapse is rate-invariant. Native variance does not increase with the factor (0.157, 0.081, 0.081, 0.068), so the operating point inflates no gap. In the paired one-sided Wilcoxon test, 13 tasks are zero-variance in both layers and drop out; over the remaining 8 the sign is unanimous, so the exact test attains its floor p = 2⁻⁸ ≈ 0.0039 at every factor. On the two decision-boundary tasks (real_integ_02, real_integ_04), residual variance is indistinguishable across layers at every factor (native 0.252–0.264, unified 0.270), as the boundary claim predicts: unification does not help decision-structured tasks.
On naturally occurring failures we claim no universal tool dominance and will state this bound explicitly. Phase 3 offers partial evidence with no scripted planner behavior: seven LLM planners (temperature 0.7 where accepted; two endpoints fall back to their default, logged per run) separate on the native layer (Kruskal–Wallis p = 0.031) but are indistinguishable under unified tools (p = 0.216), while every planner improves (+0.128 to +0.210). Gemini-3.7-flash is excluded (41.5% of its runs failed at the endpoint); the revision discloses this and reports the N = 8 bound. A study of failures in deployed pipelines is future work.
Response to W2 (Section 3.6 oracle comparison):
We accept this. The comparison changed task family and metric alongside oracle availability. E2 holds both fixed, that is adopting same four verifiable clustering tasks, same cluster-count-proximity metric, clean unified tools and varying only oracle access, via an oracle-blinded copy of the same harness (one accessor returns None; the task keeps its oracle, so scoring is unchanged). The result is : oracle-on 0.616, oracle-off 0.000, best non-oracle 0.046, showing a +0.570 within-family advantage and agreeing with Section 3.6 with family and metric fixed. Inter-harness variance (across the four within-arm harnesses' task means) falls from 0.068 to 0.0004. Blinded, the harness reduces to the adaptive planner's single-pass execution and scores identically (0.000), so the zero is attributable to the blinded plan. The oracle is consumed as a resolution search scored against ground truth, which is a minimal analogue of feedback-based refinement.
Direct answers to the question raised:
Injection-free task-quality metrics: the outcomes in Phases 4 and 6, and E1 have already showed that the decision-task residual is rate-independent. Naturally occurring failures are partially implemented via Phase 3, and a deployed-setting study is a future work.