CAES: Cost-Aware Experiment Selection via Online Shadow Prices for Autonomous Research

Research agents built on language models propose many experiments for each task, and running all of them consumes a large amount of compute. Recent methods study how to choose which experiments deserve compute. Cost-aware Bayesian optimization sets in advance the rate at which expected improvement is traded against compute, early stopping pays for a partial run of every candidate, and selectors built for research agents compare candidates only with one another. Each of these methods decides one experiment at a time, implicitly assuming that the decision does not affect the experiments that come after. In practice, compute given to one experiment is no longer available to later ones, whose value is still unknown when the current one is decided. To address this, we propose Cost-Aware Experiment Selection (CAES), which ties these decisions together through a single shadow price on compute. Given the shadow price, we frame the decision for each candidate as a Pandora’s box problem with a single box, whose exact solution chooses among skipping the candidate, running it in full, and first running a cheap partial version. The shadow price is updated online, rising when compute is spent faster than the search moves through the candidates and falling when it is spent more slowly. A parameter-free form of this update also admits a distribution-free certificate built with Learn-then-Test, which bounds the probability that the final result falls more than a chosen tolerance behind running everything. We evaluate CAES on TabRepo, LCBench, and AIRS-Bench. Given 5 to 40 percent of the compute needed to run everything, CAES outperforms ten baselines across the three benchmarks, ending closer to the result of running everything in 419 of 520 task-paired comparisons. These results show that cost-aware experiment selection with an online shadow price can guide research agents toward compute-efficient autonomous research.