- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents an LLM-based multi-agent approach to automating the optimization of preprocessing pipelines for clinical spectroscopy. The proposed system employs a proposer agent and a reflector agent, supported by a structured memory component termed the dossier, to iteratively explore and refine preprocessing configurations using feedback from previous evaluations. A multi-reward racing mechanism is incorporated to facilitate candidate selection. The framework is evaluated across 18 classification scenarios spanning four publicly available clinical spectroscopy datasets and three LLM backbones. Experimental results suggest that the approach achieves classification performance comparable to exhaustive grid search while requiring substantially fewer pipeline evaluations, and generally outperforms greedy stepwise optimization. Further ablation studies investigate the effects of different dossier components, accumulated optimization history, and LLM model size on overall performance.
Critical strengths
The paper addresses a practical and important problem in clinical spectroscopy, the selection and optimization of preprocessing pipelines. It presents a coherent and well-motivated study, with a clear connection between the practical problem, the proposed method, and its evaluation. I particularly appreciate that the authors make a serious effort to formalize preprocessing optimization as a well-defined search problem, rather than simply using an LLM as an unconstrained natural-language assistant. I appreciate the practical relevance of this work.
The experimental evaluation is also relatively comprehensive. The authors evaluate the framework across 18 scenarios involving different spectroscopic datasets and classifiers, demonstrating competitive classification performance while evaluating substantially fewer pipelines than exhaustive grid search. They further investigate the optimization process through search trajectories, LLM backbones and detailed ablation studies. the paper provides a convincing demonstration of the practical value of history-guided LLM optimization for spectral preprocessing.
Critical weaknesses
Insufficient comparison with established optimisation methods. The paper demonstrates that LLM-guided optimisation achieves competitive performance with substantially fewer evaluations than exhaustive grid search. However, the absence of budget-matched comparisons with established optimisation methods makes it difficult to assess the actual contribution of LLM-guided reasoning. Random search is a well-established baseline for hyperparameter optimisation (Bergstra & Bengio, 2012). More importantly, genetic algorithms have already been applied to Raman preprocessing optimisation (Bocklitz et al., 2011), and Brunel et al. (2021) explicitly compared genetic algorithms against random search, ant colony optimisation, and TPE in vibrational spectroscopy. Bayesian optimisation has also been applied to automated spectral preprocessing (Babatunde et al., 2025). Therefore, outperforming grid search in evaluation efficiency alone is insufficient to establish an advantage over existing optimisation strategies. Comparisons with budget-matched random search and at least one adaptive optimiser would substantially strengthen the paper's central claim.
Questions for authors
-
Could the authors compare the proposed approach with budget-matched random search and a representative adaptive optimiser (e.g., TPE or GA) to better establish the benefits of LLM-guided optimisation?
-
Appendix E (Table 13) reports pipelines violating the predefined stage constraints. Could the authors clarify how these were admitted and whether they affect the main conclusions?
-
Table 9 suggests that removing certain dossier components can maintain or improve performance. Could the authors clarify the necessity of these components, particularly the reflector agent?
4 comments
Response to Reviewer 4Qkx (2/4)
2. Stage-constraint violations
Appendix E (Table 13) reports pipelines violating the predefined stage constraints. Could the authors clarify how these were admitted and whether they affect the main conclusions?
In our framework, we deliberately did not enforce hard constraints on method placement across preprocessing stages. While method validity and parameter bounds were checked, cross-stage proposals were allowed so that we could assess how reliably agents followed the stage definitions provided in their instructions. We agree that this design choice was not sufficiently clear and have clarified it in Section 4.
Following the revised evaluation protocol, we also repeated the stage-compliance analysis using the newly generated optimisation runs and inner vs. outer evaluations. The updated results are reported in Table 17, Appendix E (previously Table 13).
Updated Table 17 — Stage-compliance summary
| Metric | DeepSeek-R1 (8B) | Llama 3.1 (8B) | Llama 3.1 (70B) |
|---|---|---|---|
| Admitted pipelines | 4,168 | 3,716 | 4,049 |
| With a cross-stage method | 66 | 51 | 0 |
| Share | 1.6% | 1.4% | 0.0% |
| Scenario-splits affected | 18/36 | 17/36 | 0/36 |
| occurrences, by family and the stage it occupied | |||
| Whittaker in baseline (registered: denoise) | 39 | — | — |
| SNV in baseline (registered: normalise) | 14 | 12 | — |
| Whittaker in despike (registered: denoise) | 7 | 15 | — |
| ASLS in denoise (registered: baseline) | 3 | 12 | — |
| IASLS in denoise (registered: baseline) | — | 9 | — |
| Vector in baseline (registered: normalise) | 5 | 2 | — |
| iModPoly in denoise (registered: baseline) | — | 2 | — |
| SavGol in baseline (registered: denoise) | 1 | — | — |
| Selected pipelines affected | 1/36 | 0/36 | 0/36 |
| Paired Δ in CV AUC (training) | −0.026 | −0.005 | — |
| Paired Δ in held-out AUC | −0.047 | −0.011 | — |
Cross-stage proposals accounted for only 1.6% and 1.4% of admitted pipelines for DeepSeek-R1:8B and Llama 3.1:8B, respectively, and none for Llama 3.1:70B. More importantly, only one of the 108 final pipeline selections contains a method outside its registered stage. This occurs for DeepSeek-R1:8B on ATR-FTIR serum without chemotherapy, with a training CV AUC of 0.828 and held-out AUC of 0.819. We report this pipeline explicitly in the new Table 18 in Appendix E, as shown below.
| Model | deepseek-r1:8b |
| Scenario | ATR-FTIR serum, no chemotherapy |
| Pipeline | |
| - despike | WhitakerHayes (kernel_size=7, threshold=15) |
| - denoise | ASLS‡ (diff_order=2, lam=1e5) |
| - baseline | identity |
| - normalise | SNV |
| CV AUC / held-out AUC | 0.828 / 0.819 |
‡ ASLS is a baseline-correction method, but the agent placed it in the denoise stage.
The paired comparisons in Table 17 show that cross-stage configurations were associated with lower AUC, both in cross-validation (−0.026 for DeepSeek, −0.005 for Llama 8B) and on held-out data (−0.047 and −0.011, respectively). Thus, these violations did not appear to improve performance. All held-out comparisons in Table 17 were computed post-hoc and were not used during optimisation or pipeline selection.
Several violations involved similarly named or related methods. For example, 'WhitakerHayes' is used for despiking and 'Whittaker' for denoising, with 22 instances of the latter being used for despiking. Other cases involved Whittaker smoothing and ASLS/IASLS baseline correction, which share similar mathematical foundations and can indeed be adapted for both preprocessing stages in practice.
Again, we emphasise that allowing cross-stage proposals was intentional, as we wanted to study how well agents follow the specified stage definitions. Strict constraints could be easily implemented, just as we already enforce parameter bounds. We agree that rerunning the optimisation with these constraints would provide a useful additional control, but the computational cost and limited rebuttal period did not allow us to complete this experiment. We hope that the expanded analysis, particularly the finding that only 1 of 108 selected pipelines was affected, helps address this concern.
Note: We observed fewer cross-stage violations in the revised experiments compared with the original results. We confirm that our pipeline generation and stage-validation logic were not changed during the revision.
Response to Reviewer 4Qkx (3/4)
3. Dossier components and Reflector Agent
Table 9 suggests that removing certain dossier components can maintain or improve performance. Could the authors clarify the necessity of these components, particularly the reflector agent?
As described in Section 3.1, our dossier builds on the meta-prompt used in Optimization by PROmpting (OPRO; Yang et al., 2024), which uses previously evaluated candidates and their scores to guide subsequent proposals. We extend this idea by organising successful and unsuccessful pipelines, performance comparisons, diagnostic feedback and task constraints into a structured record. The aim is to help agents identify promising configurations, learn from previous evaluations and avoid repeating unsuccessful choices.
We agree that not all dossier components contribute equally. As shown in Table 13 (previously Table 9), removing §7 diagnostics or the self-model (§6 and invariants) leaves mean proposal AUC essentially unchanged (0.650 vs. 0.649 for the full dossier). In contrast, removing §2 top-performing history reduces mean AUC by 0.020 ± 0.010, while removing all optimisation history leads to a larger reduction of 0.027 ± 0.012. These results highlight the importance of historical feedback, while suggesting that some supporting components offer limited additional benefit. We have expanded the rationale in Section 3.1 and the discussion of these findings in Section 5.4.
The use of a reflector agent was motivated by Nie et al. (2024), who used a separate LLM call to turn numerical evaluation results into natural-language feedback for subsequent proposals. They showed that adding an agent that provides directional feedback makes the agentic framework more robust and efficient for optimisation problems. Similarly, our reflector summarises predicted and measured AUC and sensitivity after each batch, providing feedback to the proposer through the updated dossier. We have clarified this rationale in Section 3.2.
To assess its contribution, we conducted an additional ablation on 12 optimisation runs (3 models, 2 scenarios and 2 seeds), disabling the reflector while keeping all other components and evaluation settings fixed. The results of these experiments are presented in Table 13 (previously Table 9) in Appendix E of the updated paper.
Table 13 — Reflector ablation
| Configuration | Mean proposal AUC | Best AUC | Bounds compliance | Contract compliance |
|---|---|---|---|---|
| Full dossier | 0.649 | 0.721 | 88.7% | 100.0% |
| Without reflector | 0.640 | 0.723 | 93.5% | 100.0% |
Removing the reflector reduced the mean proposal AUC by 0.009 (±0.005), with lower performance in 10 of the 12 optimisation runs. However, the best AUC remained similar (0.721 vs. 0.723), and bounds compliance improved without the reflector. These results suggest that the reflector helps improve average proposal quality, but is not essential for finding a competitive pipeline. We have added this analysis and discussed its implications in Section 5.4.
The modest contribution of the reflector also raises the question of whether simpler agent architectures could achieve comparable performance at lower computational cost. Exploring this trade-off further would be an interesting direction for future work.
Response to Reviewer 4Qkx (4/4)
Conclusion
We hope that the new budget-matched baselines, stage-compliance analysis, expanded discussions of dossier components and reflector ablations address the reviewer's concerns. We believe these additions substantially strengthen the paper and would appreciate a reconsideration of the rating in light of the revised results.
Response to Reviewer 4Qkx (1/4)
We thank the reviewer for their constructive feedback and encouraging assessment of our work. We particularly appreciate their recognition of its practical relevance and the breadth of our experimental evaluation. We address the three concerns below.
1. Comparison with established optimisers
We have added four independent baselines: random search, Tree-structured Parzen Estimator (TPE) and genetic algorithm (GA), as previously studied by Brunel et al. (2021), and Gaussian process-based Bayesian optimisation (GP-BO), as used for spectral preprocessing by Babatunde et al. (2025).
We evaluated each of the four baselines across the same 18 scenarios. To ensure comparability with our agentic methods, they use the same data splits, preprocessing method catalogue, parameter bounds and a budget of 120 pipeline evaluations per scenario. Like our agentic optimisers, these additional baselines explore continuous parameter ranges within the specified bounds, allowing direct comparisons with agentic search. We assessed each candidate using the same internal cross-validation procedure.
The results are presented in the updated Table 1, evaluated on held-out test data (see point 1 in our response to Reviewer wsKD).
Updated Table 1 — Out-of-sample AUC
Across the six datasets, our agentic approach achieved the highest mean held-out AUC in four settings and outperformed the best of the three established optimisers (TPE, GA and GP-BO) in five. We have expanded our method comparisons in Section 5.2. and added full information about the baselines in Section 4. We provide the full in-sample and out-of-sample results for individual scenarios in the new Tables 5–8 (Appendix C).
We are excited to see how future advances in LLM reasoning and agent orchestration could further improve these results. We have better highlighted this at the end of the Conclusion section of our updated paper.