- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates how different system configurations in LLM-based agents affect capability, reliability and behavior. It highlights that most existing evaluations focus on evaluating different models under fixed scaffolds, problem statements, and configuration settings. The investigation is conducted on a benchmark of four AI for Science tasks from astrophysiscs and genomics, running all possible configurations across five configuration axes (task information, reasoning mode, self-verification mode, wall-clock budget, model). The study finds that the majority (54%) of score variance comes from run-to-run noise, and that changes in task information lead to bigger changes than changes in model size or runtime budget.
Critical strengths
- The problem of evaluating agentic systems thoroughly in light of all their knobs is a pertinent problem which is often not fully considered.
- The study is considerably large with 432 configurations in total repeated 5 times each per task, taking into account in their evaluation parts of agentic systems that are not considered for ablations by related work.
- The authors intend to release all trajectories and code used to perform this study.
Critical weaknesses
- The evaluation is limited to four tasks in two domains, which is too little to make claims about the generalized impact of each "knob".
- It is not clear that the evaluated scenarios (i.e., the different configurations) represent realistic scenarios for example for how two different users would setup/use an agent for such tasks.
- The abstract claims that the Information axis has a larger effect than Budget. However, this is not the case for all models since for Claude Sonnet 5 Budget has a larger effect.
- The methodology does not consider unfinished runs, but these unfinished runs are not unfinished at random. Several of the considered knobs (Budget, Information, Model) have a direct influence on the lenght/duration/cost of runs and on whether they finish.
- Wall-clock is a potentially noisy metric. It would be better if budget is considered in terms of tokens or token cost.
I think the paper's goal is worthy but doing the study on such a small number of tasks reduces the relevance of results since it's quite unclear whether the results generalize. Without doing it on more tasks, the main takeaway is that there are many knobs and we should consider them when evaluating agentic systems (which is not necessarily novel).
Questions for authors
see weaknesses
Response to authors
I'll admit I'm still not totally sold on this paper. I think my biggest sticking point is still Q2. But I accept that the authors have done a thorough job of addressing my other concerns so I'll raise my score to a weak accept.