acceptodds
Under review as a conference paper at ICLR 2027

WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents

Abstract

AI research agents must predict the effects of computational changes after budgeted experiments. WhatWorkedBench evaluates this experimental understanding through a delivered response surface of configuration scores. Agents inspect workflow code and buy measurements; exhaustive CPU references score conditional component effects, mean pair interactions, configuration choice, and delivery. The catalog spans 36 task conditions, 30 sources, 8 workflow families, and 1248 indexed configuration records, with 4,206 numerical controls. At eight purchased measurements plus two free anchors, pair-effect ridge selects an exact optimum on 15 of 22 sources; 13 of these cases have at least one conditional-effect error exceeding 10% of the task utility range. Shared-estimator comparisons measure acquisition and reconstruction on common observations. In a prospective typed study on 12 four-factor sources, DeepSeek Flash submits 12/12 direct tables and gains 0.147 recovery over a Gaussian process (GP) fitted to the same observations. Pro delivers 11/12 artifacts, with an all-attempt GP difference of −0.001 and a delivered-only difference of +0.063. On six prespecified new agent-evaluation sources, Flash and Pro gains are 0.149 and 0.074. In eight typed six-factor episodes, seven final tables satisfy verified code equivalences. Four fresh agents pass all six registered rules through named estimators and deliver consistent tables at 0.682 recovery versus 0.710 for separate direct-table runs. WhatWorkedBench links acquisition, inference, program structure, and delivery.

Primary area:
datasets and benchmarks
Keywords:
AI agents, experimental design, scientific discovery, agent evaluation, benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BrrpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces WhatWorkedBench, a benchmark in which an agent inspects workflow code with four or six binary options, buys a limited number of measurements, and submits a full table of predicted scores. Submissions are graded against exhaustively executed references on how well they recover each option’s effect in every background, alongside pair interactions, best-configuration choice, and delivery. The catalog covers 36 task conditions over 30 sources and 8 classical CPU workflow families. Validation consists of ridge and Gaussian-process controls plus episodes from two DeepSeek models, with shared-estimator comparisons separating acquisition from inference.

Critical strengths
  1. The motivation is sound: predicting how component changes behave in new backgrounds is a real requirement for research agents, and the paper shows it is not captured by picking the best configuration.
  2. The method design is careful, with exhaustive references for every conditional effect and shared-estimator comparisons that separate acquisition from inference.
  3. The experiments are solid and transparently reported, covering thousands of numerical controls, frozen plans, and separate all-attempt and delivered scores.
Critical weaknesses
  1. The evaluation scenario is narrow, limited to four or six binary switches on deterministic classical CPU pipelines.
  2. “Experimental understanding” is too broad a name for what is measured, which is response-surface prediction on small binary grids, and the authors should narrow the framing accordingly.
  3. The paper is difficult to follow, with many poorly defined terms and a main conclusion that is hard to identify. I encourage the authors to address these issues of clarity and organization in the submitted version.
Questions for authors
  1. The benchmark lacks diversity in its option types, since every factor is a binary switch; would it be feasible to add at least one workflow family with non-binary options (e.g., multi-level or continuous hyperparameters), and do the main findings still hold there?
  2. Since full-code and opaque views score almost the same, what evidence shows the metric captures understanding of the workflow rather than numerical interpolation?
Reviewer_M4JhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes WhatWorkedBench, which tests whether an agent can predict the effect of each workflow option after a few budgeted experiments. The benchmark covers 36 tasks from 30 public sources across eight workflow families, including UCI tabular datasets, time series, images, BEIR retrieval corpora, MIT-BIH ECG records, and small social graphs. Each task has four or six binary options, and the authors run every configuration to get a full reference. They test DeepSeek V4 Flash and Pro against ridge regression and Gaussian process (GP) baselines, and find that the agents pick experiments about as well as a simple baseline, and predict better than the GP only under a later interface.

Critical strengths

The question itself is interesting. Checking whether an agent actually understands what each change does, not just whether it finds the best config, is a useful angle. But the setup isn’t rigorous enough to back up real scientific findings.

Critical weaknesses
  • This paper is really hard to read. The writing is difficult to follow. Many phrases are vague, and the paper uses many terms without clear definitions. The results sections mostly list numbers from many cohorts with different prompts and interfaces. It is difficult to find the main conclusion, and some results from different cohorts do not agree.
  • The agent experiments are too small. Most comparisons use few sources and few runs, and the paper tests only two models from one model family. This is not enough to support general claims about AI agents.
  • I am also concerned about how stable the experimental process is. At the main budget, the agents beat the GP baseline only in a later study that uses a new interface. On top of this, API rate limits and timeouts cause several failed runs, which pull the reported scores down.
  • Some conclusions are not reliable. The comparison between the agent and the GP baseline is not fair, because the agent gets the code and a Python sandbox but the GP gets only the measured scores. The acquisition results also change when the estimator changes, and the paper gives no upper bound for good acquisition.
Questions for authors

See the weakness section above

Reviewer_CABgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper evaluates how well AI agents choose sequences of parameter settings to try that eventually lead to a good/optimal settings in any context. They argue that this should be evaluated by measuring how well agents predict the effect of changing each parameter in every combination of the others.

Their setup is as follows:

  • Each task is a small ML or signal-processing pipeline with 4 or 6 binary parameters (switches), and the agent is given the pipeline’s code.
  • The agent gets the all-off and all-on scores for free and can “buy” a fixed budget of additional configurations to test.
  • The agent must then predict the output score for every configuration.
  • Predictions are graded on the error in the predicted change in score from flipping each parameter, in every combination of the other parameters.

The agents are compared to non-agentic approaches. These fit standard regression models (ridge regression and Gaussian processes) from parameter settings to score, and choose which configurations to buy by targeting those where the model is least certain.

Results:

  • On small tasks (4 switches, 16 configurations), DeepSeek Flash with the revised prompt beat the regression baselines, even when the baselines were fitted to Flash’s own measurements.
  • The agents’ advantage came from predicting the scores of unmeasured configurations, not from choosing better configurations to measure. Their choices were no better than a simple fixed design.
  • Both agents and baselines could often identify the best configuration while still badly mispredicting at least one change error.
  • On larger tasks (6 switches, 64 configurations), results were mixed. With the revised prompt, on four tasks whose code reveals switches that have no effect in certain settings, Flash beat standard regression on the same measurements and about matched regression that was told those rules. With the original prompt, and on two additional task families (heartbeat detection and graph link prediction), Flash did worse than a Gaussian process fitted to its own measurements, even though on the latter it had measured half of the configurations.
Critical strengths
  • AI agents are increasingly used to optimize hyperparameters, so it matters whether they can gather information from a budgeted number of evaluations to find the best parameter setting. The paper evaluates the ability of DeepSeek agents (Flash and Pro), on several small ML and signal-processing tasks.
  • The paper compares agents' performance to standard statistical methods: ridge regression and Gaussian processes.
  • The paper performs an interesting ablation to determine whether the selected “buys” or the scoring given these measurements drove success, two skills that are usually tangled together. Holding the prediction method fixed and swapping in different agents’ or baselines’ measurements isolates how good the choice of experiments was. Holding the measurements fixed and swapping prediction methods isolates how well the results were used. This revealed that the agents’ advantage came almost entirely from predicting unmeasured scores, while their choice of what to measure was no better than a simple fixed design.
Critical weaknesses
  • The experiments appear to be not well-controlled. The paper names only the LLM API identifiers (deepseek-v4-flash, deepseek-v4-pro), which are aliases DeepSeek updated during the study period. It does not seem like the work freezes the LLM version, so likely some runs use different versions of the model. Table 3 reports “original” and “additional” runs separately without explaining what distinguishes them. Since the prompts and tools are described as the same, this raises the question of whether the underlying model changed between batches.
  • Several Pro runs failed to submit after repeated rate-limit errors from the provider, and these count as zero, so model performance is mixed with API reliability. These should be excluded.
  • Each task is a small CPU pipeline (logistic regression, ridge, k-means, TF-IDF, scipy filters) on small datasets, with 4 or 6 binary switches. Continuous parameters such as regularization strength are reduced to two values. This makes exhaustive ground truth possible, but it is far from the settings that motivate the work, such as expensive training runs with continuous hyperparameters and many factors. It’s unclear whether the results would carry over.
  • This paper was exceptionally hard for me to parse and understand. It introduces many non-standard, loosely defined terms (“background,” “typed cohort,” “delivery,” “family-macro,” “shared-estimator cross-comparison”), and uses them before defining them. It reuses “original” for both a set of tasks and a batch of runs, and it reports numerous sub-studies whose setups differ in ways that are hard to track. The abstract lists results from about six studies without explaining what distinguishes them.
  • The early runs offered ridge-regression helpers as an explicit submission option, and agents always used them, which is not standard agent behavior when tuning hyperparameters, thus I think these "original" prompt experiments were not very informative. I think the paper would be better off excluding somewhat failed experiments, maybe just including the prompt iteration in the Appendix, but it was very hard to keep track of what each experiment consisted of, and what was comparable.
  • The heartbeat and graph runs don’t state which prompt was used, thus it's hard to know how to compare them.
  • Most agent results rest on 4-12 tasks with one to three runs each, and many of the reported bootstrap ranges include zero. Only one model family (DeepSeek) is evaluated, so it’s unclear whether the findings reflect agents in general.
Questions for authors
  • What model version(s) were used? Was it pinned across experiments? What distinguishes the “original” and “additional” batches in Table 3, and why are they reported separately rather than pooled?
  • What exactly changed between the original and revised prompts?
  • Which prompt and setup were used for the heartbeat and graph runs?
  • Why does Flash consistently outperform Pro?
  • Is there any evidence the findings hold for larger tasks: more switches, continuous parameters, or costlier training runs?
  • Have you evaluated model families other than DeepSeek?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.