- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces WhatWorkedBench, a benchmark in which an agent inspects workflow code with four or six binary options, buys a limited number of measurements, and submits a full table of predicted scores. Submissions are graded against exhaustively executed references on how well they recover each option’s effect in every background, alongside pair interactions, best-configuration choice, and delivery. The catalog covers 36 task conditions over 30 sources and 8 classical CPU workflow families. Validation consists of ridge and Gaussian-process controls plus episodes from two DeepSeek models, with shared-estimator comparisons separating acquisition from inference.
Critical strengths
- The motivation is sound: predicting how component changes behave in new backgrounds is a real requirement for research agents, and the paper shows it is not captured by picking the best configuration.
- The method design is careful, with exhaustive references for every conditional effect and shared-estimator comparisons that separate acquisition from inference.
- The experiments are solid and transparently reported, covering thousands of numerical controls, frozen plans, and separate all-attempt and delivered scores.
Critical weaknesses
- The evaluation scenario is narrow, limited to four or six binary switches on deterministic classical CPU pipelines.
- “Experimental understanding” is too broad a name for what is measured, which is response-surface prediction on small binary grids, and the authors should narrow the framing accordingly.
- The paper is difficult to follow, with many poorly defined terms and a main conclusion that is hard to identify. I encourage the authors to address these issues of clarity and organization in the submitted version.
Questions for authors
- The benchmark lacks diversity in its option types, since every factor is a binary switch; would it be feasible to add at least one workflow family with non-binary options (e.g., multi-level or continuous hyperparameters), and do the main findings still hold there?
- Since full-code and opaque views score almost the same, what evidence shows the metric captures understanding of the workflow rather than numerical interpolation?