- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduce SciProdBench, a benchmark to assess models' abilities to synthesize literature and execute code across 184 scientific tasks (biology, chemistry, ...). Tasks are constructed from existing research papers and repositories, with rubrics used to evaluate whether the model completed the task successfully and includes all deliverables/evidence for the models' claims. The authors' initial evaluations show coding agents excel in instruction-following but struggle with evidence synthesis, motivating a new workflow called EulerFlow: an agent that reports findings, checks them with existing evidence, and refines the unsupported ones. EulerFlow improves existing harnesses on SciProdBench.
Critical strengths
- The dataset would likely help other researchers who are interested in evaluating scientific execution and synthesis tasks. I like the split of tasks that require just literature search, or also literature search and code execution
- The main idea of EulerFlow is interesting: generate proposals, cross-check them with evidence, and revise unsupported ones. I think the harness contribution could be reframed, but I see how this could be useful in a variety of agentic tasks, not just AI4Science
Critical weaknesses
- I didn't see any examples of the benchmark tasks in the paper, so it's hard to understand what the dataset looks like. I would recommend adding a full example of the input task, rubrics for scoring, and full trajectory of a model completing such a task, even if it's in the appendix. Otherwise, I really can't tell whether SciProdBench is truly testing the capabilities that it claims to. Figure 1 describes tasks at a high level (i.e, the steps that are taken) but an actual example would be helpful
- The dataset lacks quality control, so it's hard to tell how rigorous the dataset construction is. It would be great if the authors could design a human annotation setup to quantify what proportion of the tasks are high-quality, and what should people look out for when using this dataset
- The proposed modeling contribution of EulerFlow is weak: its description in section 4 is sparse and implies that it is implemented as a skill file, so I'm not sure if it constitutes an entire harness. It's also unclear how much this harness is engineered towards the synthetic tasks that the authors generate. I would recommend testing generalization on another dataset, and tempering claims about developing a new harness in this paper. I also suspect this process is much more computationally expensive since the model needs to check and revise each generated output
Questions for authors
- The approach of representing tasks as graphs and then prompting LLMs to generate descriptions, test cases, etc. from the graphs seems unnecessarily complex. Is there any advantage to this approach that the authors could argue, theoretically or empirically?
- Can the authors provide quality control statistics of the dataset with human annotations?
- Is there any way to gauge the upper bound of this task? In theory it's 100%, but it might be useful to use the quality control statistics from (2) or discuss how humans could complete some of these tasks to establish a stronger ceiling
- Can the authors provide examples of the tasks in the dataset?
- What is the cost, generalization, and novelty of EulerFlow?