acceptodds
Under review as a conference paper at ICLR 2027

SLDBench v2: Active Scaling Law Discovery

Abstract

Scaling laws predict large-model performance from small experiments, but finding a reliable law itself requires costly experiments. SLDBench tests whether language-model agents can discover scaling laws when all experimental results are given upfront. We introduce SLDBench v2 with two extensions. First, we expand the corpus from 8 to 127 scenarios spanning language-model pretraining, downstream performance, and vision–language transfer, and select 30 challenging scenarios on which a reference power-law fit reaches R² ≤ 0.8; they represent at least 1.09 × 10²³ FLOPs of source-reported training compute. Second, we introduce active scaling law discovery: an agent chooses which experiments to run, pays each one’s recorded cost (e.g., training FLOPs), and revises its law as outcomes arrive, and is scored on the same hidden extrapolation targets as in the passive setting. Across 7 frontier agent systems, median R² drops from 0.72–0.75 (passive) to 0.24–0.56 (active), and prediction error rises 1.5–4.7×, depending on the system. Agents favor cheap experiments, spending 9–44% of the full-pool cost on average; the systems that spend least fail most often, and cheap purchases leave experimental conditions needed for extrapolation unobserved. Experimental planning thus remains a key open challenge for agents that assist model development in the prevailing AI-for-AI study.

Primary area:
datasets and benchmarks
Keywords:
language model agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uQcpOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper builds on SLDBench, which asks LLM agents to discover a scaling law from given data that can predict the performance of larger runs. They add to this by collecting 127 scenarios (~3.4 million records) and selecting 30 tasks from 14 sources that satisfy R^2 <= 0.8 using a reference power law in both settings. They also add an Active setting where the agent is able to buy experiments from previously recorded runs (spending is recorded and signed, with a separate verifier) and is tested on the same hidden tests as the Passive setting. Cost is calculated by taking the cost of the experiments purchased divided by the cost of all available experiments and is reported but not used to determine the score. They find that all 7 systems (3 repeats) have medians in the Passive setting between 0.72 and 0.75 and in the Active setting between 0.24 and 0.56. All systems perform 1.5-4.7 times worse (MSE) when they have to purchase their data and systems that spent the least amount had the highest increase in failures. They also find that using a fixed scaling law and purchasing data that spans the prices of each group performs better than purchasing the cheapest experiments from each group.

Critical strengths
  1. Adding an Active setting to SLDBench is a good idea and it is implemented carefully. By allowing agents to purchase previously run experiments, the authors are able to test if agents can effectively choose what data they want to collect to achieve their goal. By replaying previous runs they allow these types of questions to be tested more cheaply and reproducibly. Having a root-only oracle along with a signed ledger and a separate verifier allows the agent to be checked on its actions. Since both Active and Passive share the same testing data, the authors can directly compare how much better an agent is at discovering scaling laws without data collection. I believe with the addition of 127 scenarios this will be a helpful benchmark for future agents.

  2. All systems perform worse when they have to purchase their data. Error increases ranged from 1.49x to 4.71x with all 95% confidence intervals above 1 in figure 3c. The authors also show that using a fixed scaling law and purchasing data that covers the prices of each group seems better than purchasing the cheapest points from each group.

Critical weaknesses
  1. When discussing agent behavior they should mention that the agent is instructed to spend as few FLOPs as possible (Task T22 Prompt in Figure 2c). On line 177 we see the prompt that states this. Lines 214-215 show that cost is not taken into account when scoring the agent. The text only mentions that the prompt informs the agent that spending is being tracked on lines 205-207. If this is stated in all prompts then lines 21-22 and 311-317 should mention this when stating that agents like cheaper experiments and systems that spent less failed the most.

  2. Systems are not clearly differentiated by the Active setting. Line 254 states that Active SLD is able to differentiate systems. Table 5 shows that systems all have similar percentages of Active runs with an R^2 greater than 0.5 (41-51%). Kimi K3 is also able to achieve the highest median because they are not including runs that don't have a score. There are 41 out of 90 runs that have an R^2 greater than 0.5, which with a median of 0.555 would mean at least 8 runs did not have a score. They already state on lines 362-363 that system medians are not a precise measure. So the statement that Active SLD is able to differentiate systems should be modified to reflect that. Also the reference law used to generate predictions clips its values to the min/max of the training target values (line 647). Since this is used to determine which tasks to use and in the non-agent baselines, they should mention this in the main text.

These are all things that could be easily addressed in the wording of the paper, so I lean towards acceptance.

Questions for authors
  1. Was "Spend as few FLOPs as you can." included in all of the task prompts? How do you think this affects the agents' spending?
  2. How many runs per system in the Active setting did not have a score, and why?
  3. Does the agent know what it's trying to extrapolate to or the input of the test set? If not, how do they know what experiments to buy? (line 355)
  4. Does the clipping mentioned on line 647 happen for each group or overall? How do you think this affects the task selection and section 3.4?
  5. Which tasks price checkpoints at the cumulative cost of their run (line 713)? What are the "task-specific columns" for Passive only? (line 708)
  6. Do you think agents may know scaling laws that have been published on these tasks? Will you be releasing the agent trajectories and all 127 scenarios?
2 comments

We thank the reviewer for recognizing the Active setting and its careful implementation, and for the thoughtful comments. We address the points in turn.

Systems are not clearly differentiated by the Active setting. Line 254 states that Active SLD is able to differentiate systems. Table 5 shows that systems all have similar percentages of Active runs with an R^2 greater than 0.5 (41-51%). Kimi K3 is also able to achieve the highest median because they are not including runs that don't have a score. There are 41 out of 90 runs that have an R^2 greater than 0.5, which with a median of 0.555 would mean at least 8 runs did not have a score. They already state on lines 362-363 that system medians are not a precise measure. So the statement that Active SLD is able to differentiate systems should be modified to reflect that.

Active SLD still separates systems far more than Passive SLD, and lines 362–363 do not contradict this. In Passive, the seven medians span only 0.723–0.753; in Active, they span 0.235–0.555, and each system's error grows by a different factor, from 1.49× to 4.71×, with every 95% interval above one (Figure 3c), as the reviewer also notes. Although the share of Active runs above 0.5 is similar across systems (41–51%), the share of failed runs is not (21.1–40.0%), and for extrapolation it is precisely these failures that would mislead a decision about a larger run.

Lines 362–363 concern a finer question: the order of adjacent systems whose medians are close. Kimi K3's first place, which depends on how unscored runs are counted, is such a case. It does not change our conclusions.

That said, we fully agree that a claim about separating systems should be stated with care, so that it is not read as a precise ranking. We have therefore revised this statement (Section 3.2) and report the unscored runs and the resulting medians in Appendix B.1.

Q1.

Was "Spend as few FLOPs as you can." included in all of the task prompts? How do you think this affects the agents' spending?

Yes. All 30 Active task prompts include the sentence “Spend as few FLOPs as you can.” Removing this request in a reduced ablation raised median spending from 0.64% to 27.88% of catalogue cost for Qwen3.8 Flash and from 39.97% to 63.52% for DeepSeek V4 Flash.

Table U1. Median spending and failures with and without the cost instruction (six runs each; Appendix C.1)

ModelSpend with instructionSpend without instructionFailures with instructionFailures without instruction
Qwen3.8 Flash0.64%27.88%64
DeepSeek V4 Flash39.97%63.52%55

We report R² and experiment cost separately. The revision states this instruction in Section 2, gives the full text in Appendix A.5, and discusses its effect in Section 3.2.

Q2.

How many runs per system in the Active setting did not have a score, and why?

Table U2 covers 90 scheduled Active runs per system. Nine Kimi K3 runs and three Hy4-Preview runs timed out. Every run of the other five systems returned a finite score. Table 1 takes the median of finite scores. Table 5 (Table 7 in the revision) uses all 90 runs as its denominator.

Table U2. Active runs without a finite score

SystemUnscored / scheduled runsCause
Kimi K39 / 90Timeout
Hy4-Preview3 / 90Timeout
DeepSeek V4 Pro, DeepSeek V4 Flash, GPT-5.6 Sol, Gemini 3.8 Flash, Qwen3.8 Flash0 / 90 eachNone

Table 1 pools the scored runs from all three rounds before taking the median. By that measure Kimi ranked first at 0.555. Ranking unscored runs last yields 0.405, third behind DeepSeek V4 Pro and GPT-5.6 Sol. The seven shares above 0.5 are close, ranging from 41% to 51%.

Q3.

Does the agent know what it's trying to extrapolate to or the input of the test set? If not, how do they know what experiments to buy? (line 355)

Agents know the target and extrapolation direction, but do not see the test configurations while buying. They see candidate inputs, groups and prices. T07 asks for normalized PIQA accuracy at higher token counts. An agent could buy two token scales for one model, observe how accuracy changes, then decide whether to buy a larger scale or probe another model group. The verifier supplies hidden inputs and group IDs to the submitted law only at scoring. We have added this to Appendix A.3.

Q4.

Does the clipping mentioned on line 647 happen for each group or overall? How do you think this affects the task selection and section 3.4?

We reapplied the original selection criterion without clipping and selected exactly the same 30 tasks. We use this simple safeguard only in the reference law to limit extreme extrapolations. Within each task, all groups share one pair of bounds derived from the measurements used to fit the law.

Suppose the language models used to fit the reference law have validation losses ranging from 4 to 6. The difference is 2, so extending the range by 2 at each end gives bounds of 2 and 8. If the law predicts a loss of 1 for a larger model whose measured loss is 3, we raise the prediction to the lower bound of 2 and score it against 3.

For Section 3.4, we kept each strategy’s purchases and fitted law fixed and scored its predictions both ways. At all seven partial budgets, buying at different prices within each group gave higher median test R² than buying in ascending price order, with or without clipping. Table U3 shows the 2% budget. We have stated this rule in Section 2 and described it in full in Appendix A.2.

Table U3. Median test R² at a 2% budget cap

Selection within each groupWith clippingUsing raw predictions
Buy in ascending price order−1.456−2.040
Buy across the price range−0.0050.015

Q5.

Which tasks price checkpoints at the cumulative cost of their run (line 713)? What are the "task-specific columns" for Passive only? (line 708)

Tasks priced by cumulative training cost per checkpoint include T01 to T12 from ColPret, T16 from Gemstones, T23 on loss transfer, T24 from Misfitting, T27 from OpenWebText and T28 from ProteinLM. Translation task T29 uses 6 × parameters × cumulative examples seen. Two checkpoints from one run are charged separately. Revisiting the same checkpoint is free.

The extra Passive training fields vary by task. On T01 they are model name, training data, tokenizer, effective parameters, reported training tokens, source row and split. T22 includes the original mIoU, with 1 − mIoU as its target. Submitted laws receive the same test inputs and groups in both settings. We have added these details to Appendix A.3.

Q6.

Do you think agents may know scaling laws that have been published on these tasks? Will you be releasing the agent trajectories and all 127 scenarios?

During evaluation, agents have no network or web search, so they cannot look up source papers, and they never see the hidden test targets. Prior exposure during pretraining is still possible, since the source papers and their laws are public. As in SLDBench, we therefore treat SLDBench v2 as a literature-aware benchmark: like a human researcher building on prior work, an agent may draw on published laws, and the benchmark measures whether it can turn that knowledge into accurate extrapolation from the data it sees or buys. Recalling a published law also helps less here: the 30 tasks were selected because a standard power law extrapolates poorly (R² ≤ 0.8), and in the Active setting a known law form does not tell the agent which experiments to buy or when to stop.

We will release all 127 scenarios, including the 60 with priced experiment catalogues, and the full trajectories of all reported agent runs, including conversations, tool calls, purchases and costs where applicable, and the submitted laws. The revision adds this discussion to Appendix A.3 and the release plan to the reproducibility statement.

Reviewer_YdfCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends SLDBench in 2 ways. Firstly, it expanded scaling-law corpus and selected 30 hard tasks from 14 sources. In addition, it introduced an active setting: agent purchase experiments from a priced catalogue of recorded runs, and return fixed laws with results, and finally evaluate in a hidden large-scale run that has the same setting with passive ones. In 7 frontier agentic systems, the median R2R^2 is almost the same under the passive settings, and all declines in active settings with huge gaps. The authors attribute this loss to what agents purchased, instead of sequential access itself. In other words, agents prefer cheap experiments, system that costs the least fails the most, and cheap purchases make the conditions required for extrapolation never to be observed.

Critical strengths
  1. Coupling law construction with price-aware experiment selection is a well-motivated setup in AI4AI research. The replay design makes this setup low-cost, reproducible, and auditable: both setups share the same hidden test set, and every purchase is recorded in a signed ledger. The analysis is also thorough and clear.
Critical weaknesses
  1. Compared with related works, the contribution is incremental. From a conceptual perspective, this benchmark simply combines the open-ended law discovery with cost-aware selection from finite priced pools, and for the latter one we have SL2SL^2 that already studied SLDBench tasks from the condition of a given law family. The expanded corpus contributed 30 tasks, but on these tasks the passive settings cannot differ current systems. Since this paper didn’t compare with SL2SL^2 and SLDAgent, the readers cannot tell what this combination reveals that single components don’t.
  2. There is a confounding factor regarding the core conclusions on experimental planning. The agent was required to expend as few FLOPs as possible, but accuracy and cost were reported separately; there was neither a shared budget nor a shared accuracy target, so each system may have landed at a different trade-off point. Therefore, low expenditure and the resulting failure does not necessarily indicate poor planning ability. As the paper itself acknowledges, its non-agent baseline cannot indicate whether the agent selected data efficiently. Moreover, since no adaptive methods are run under comparable budget, active gap cannot be differed from the difficulty gained from extrapolating from partial catalogues itself.
Questions for authors
  1. If explicit budget upper bound is set, or the command on try to spend less is removed, how will the results change? Does the correlation between low spending and failure still exists?
  2. How does SL2SL^2 (under active settings and comparable budget) and SLDAgent (under passive settings) perform on these tasks? To be clear, if we give all the data purchased from different strategies under the same budget to the same agent or the same fitting process, how much will active gap remain?
  3. The paper has already used source-level bootstrap to error multiplier, can we expand it to active median of every system and its ranking?
5 comments

We thank the reviewer for the thoughtful comments and for engaging with our work. We address the points in turn.

W1.

The contribution is incremental over prior work. Conceptually, the benchmark combines open-ended law discovery from SLDBench with cost-aware selection from finite priced pools, which SL2\mathrm{SL}^2 already studies on SLDBench tasks with a supplied law family, and the expanded corpus contributes 30 tasks on which the passive setting no longer separates current systems. Because the paper compares against neither SL2\mathrm{SL}^2 nor SLDAgent, it is unclear what this combination reveals beyond what each component has already established.

We respectfully disagree that SLDBench v2 is a combination of SLDBench and SL². SLDBench gives the agent all the data and asks for a law; SL² gives it a law family and a budget and asks which experiments to run. In Active SLD the agent has neither: it must choose experiments and construct the law at the same time, and the two choices depend on each other. Whether an experiment is worth buying depends on which laws the agent is still considering, and the law it ends up with depends on what it bought. The SL² paper itself lists misspecified law families and long-horizon budgeting as open problems; our tasks were selected because a standard power law fails on them, and Active sets no budget.

Neither SLDBench nor SL² evaluates an agent that must both select experiments and construct the law. In our results, Passive ranks barely predict Active ranks (Spearman 0.05), so finding good laws from complete data does not make an agent good at Active SLD. Failures also arise when the law an agent assumes decides what it buys. On T22, one agent assumed that all groups share one exponent, and therefore never bought the larger runs that could have tested it; those groups account for 96.4% of its test error. This is also why the saturation of Passive on these tasks (0.723–0.753) matters: the spread in Active (0.235–0.555) comes from having to choose the data, not from fitting laws alone.

We therefore believe SLDBench v2 is not incremental: it measures how well an agent gathers evidence and builds a law together, which neither SLDBench nor SL² can measure.

That said, we fully agree that comparisons with existing methods are important. We have expanded this discussion in Sections 2 and 4, and added the comparisons in Q2 below to the main text (Table 2, Section 3.4).

W2.

The central conclusion about experimental planning is confounded. Agents are told to spend as few FLOPs as they can, but with accuracy and cost reported separately and no common budget or accuracy target, each system may settle on a different trade-off, so low spending and the resulting failures need not indicate poor planning. As the paper acknowledges, its non-agent baselines cannot show whether agents select data efficiently, and because no adaptive method is run at comparable cost, the active gap cannot be separated from the intrinsic difficulty of extrapolating from partial catalogues.

We respectfully note that this concern may partly stem from a misunderstanding of the task. Active SLD evaluates an agent's ability to plan its own experiments. We quote the task prompt here (full text in Appendix A.5): "Spend as few FLOPs as you can. … buying too little leaves the law unidentified. Choosing which experiments are worth their price is the task." It asks the agent to judge for itself when the data suffice, not simply to spend as few FLOPs as possible or to trade accuracy for cost.

… so low spending and the resulting failures need not indicate poor planning.

In most cases, low-cost failures do reflect poor planning. We randomly sampled 30 scored failures that spent under 5% of the pool and read why each agent stopped buying. Of the 19 that gave a reason, 14 considered their data or fit sufficient, for example after a held-out check on the cheap data they had bought; only 3 mentioned cost. For example, DeepSeek V4 Flash stopped at 2.01% after noting "RMSE 0.305 on held-out data … suggests the model generalizes well", then scored R² = −15.6. These failures therefore still support our core conclusion that experimental planning remains a key open challenge.

The partial catalogue is also the agent's own choice, since Active sets no budget cap. Overall, the failure rate is about 55% below 1% of the pool, but about 25% both at 5–20% and when the whole catalogue is bought. Within the 21 tasks with runs on both sides of 5%, the median failure rate is 46% below 5% and 12.5% above it (p < 0.001). Extrapolating from part of the catalogue is therefore not what drives the gap; stopping too early is.

That said, we fully agree that how planning is evaluated under the cost instruction should be stated clearly. We have added the full prompt (Appendix A.5) and these run-level analyses (Section 3.2, Appendix C.1) to the paper.

Q1.

How do the results change under explicit budget caps, or without the instruction to minimize spending? Does the link between low spending and failure persist?

Without the instruction, agents spend much more, yet most runs still fail. Spending rose for both models, yet DeepSeek V4 Flash still failed in five of six runs and Qwen3.8 Flash went from six failures to four (Table Y1). In the 630 original Active runs, runs that spend more on the same task usually score higher (on 27 of 30 tasks). The link between low spending and failure therefore persists, but spending more does not by itself remove failures; the failures ultimately reflect the agent's own limited planning ability.

Table Y1. Median spending and failures with and without the cost instruction (six runs each; Appendix C.1)

ModelSpend with instructionSpend without instructionFailures with instructionFailures without instruction
Qwen3.8 Flash0.64%27.88%64
DeepSeek V4 Flash39.97%63.52%55

We keep the instruction because deciding when to stop is part of the task; the revision states it in Section 2 and gives the full text in Appendix A.5.

Q2.

How do SL2\mathrm{SL}^2 (in the active setting, at comparable cost) and SLDAgent (in the passive setting) perform on these tasks?

Both perform on par with or below the agents we evaluate. For SL², under DeepSeek V4 Pro's actual costs on 12 tasks and with the same law, the model's points score 0.560 and SL²'s 0.520.

For SLDAgent, we ran it in Passive. SLDAgent reaches a median R² of 0.773, within the 0.735–0.776 range of the seven systems on the same tasks. In Passive, general agents thus already discover laws as well as a dedicated method. Active additionally requires them to plan their own experiments while discovering the law, and their large drop there comes from this added requirement, which SLDAgent cannot meet, since it has no mechanism for choosing experiments. We have added these comparisons to Table 2 (Section 3.4), with the settings in Appendix B.4.

More directly, if data bought by different strategies at the same cost were given to the same agent or fitting procedure, how much of the active gap would remain?

At the agents' actual costs, a different selection strategy leaves most of the gap. We capped each task at the cost of Qwen3.8 Flash's and Kimi K3's first Active run, gave the model's points and randomly chosen points to the same reference fitting procedure (Appendix A.2), and scored all 30 tasks.

Table Y2. Median test R² of the same reference fitting procedure under the agents' own cost caps

Cap source (median cap)Model's pointsRandom pointsWhole catalogue
Qwen3.8 Flash (6%)0.0160.0500.455
Kimi K3 (16%)0.1560.3000.455

At these costs, every selection stays 0.16–0.44 below the whole catalogue. The Active gap therefore depends mainly on how much the agent buys and when it stops, which is the planning ability we evaluate.

Q3.

Can the source-level bootstrap already used for the error multipliers be extended to the per-system active medians and their ranking?

Yes, and the main findings of the paper remain robust under the source-level bootstrap. We resampled the 14 sources with replacement 100,000 times, keeping every task, repeat and system pairing within each sampled source, and recomputed each system's medians and ranks.

Table Y3. Share of source-level resamples in which each finding holds

FindingShare of resamples
Systems are spread further apart in Active than in Passive99.9%
All seven systems score lower in Active than in Passive97.1%
The top three systems (Kimi K3, DeepSeek V4 Pro, GPT-5.6 Sol) score above the other four as a group99.6%

Adjacent systems can swap places, but the three strongest systems stay above the other four as a group. We have added the per-system intervals to Table 1 and these results to Appendix B.2 (Table 8).

Thanks for the detailed response. The added source-level bootstrap and the stopping-reason analysis solved my Q3 and partially responded to W2. But I still have 2 concerns. First, the full prompt in Appendix A5 clearly tells the agents that submissions are compared as an accuracy-versus-compute trade-off, while the scoring only uses R2R^2, so the Active ranking still reflects how each system handles this unspecified trade-off. And the ablation of removing this prompt only contains 3 tasks and 6 runs per condition, and failures are counted based on reference law instead of the agents’ own law, so the conclusion that failure comes from planning ability is questionable. In addition, Table Y2 shows that under the same fitter and budget, random points outperform agents’ own selection, so what is bought matters as much as how much is bought, which also weakens the claim that CARBS and SL2SL^2 are at best comparable to agents, since Table 2 refits the law form the agent chose after seeing its own purchases. I would therefore keep my assessment of weak rejection.

Reviewer_R7LCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new benchmark for the active discovery of scaling laws and evaluates several agent-based approaches on the benchmark. The benchmark is constructed from public data covering several different large scale pipelines and problem settings.

Critical strengths

Targets a very important and timely problem of guided discovery of scaling laws that will only become more relevant in the future. Benchmarks are critical to establish in new problem settings, so this paper is of very high relevance to the ICLR audience and I expect several future works to use this benchmark.

Critical weaknesses

As benchmark papers set the standard for evaluation protocols, the experiments for this paper need to be especially well designed. However, there are several crucial flaws. Experiments are run with three seeds only and the error in the measurement is not quantified in the main results (e.g., Table 1), the authors themselves mention an alarming difference across repeats ("Outcomes vary across repeats." paragraph). Also the random baselines are omitted in the main results and I am wondering about several other points (see my questions).

Questions for authors
  1. Why categorize scenarios as eligible or not in terms of difficulty? Would it not be interesting to consider the scenarios with R2 > 0.8?

  2. Why remove uninformative features at the level of the benchmark design? Is it to help the search method?

  3. Why would a method like CARBS (Fetterman et al.) not be applicable to your setting? Is it only because it is supplied a cost-cap? If so, why not benchmark these kind of methods for a couple of pre-set costs? May that be a strong baseline?

  4. Why not compare to a referemce law and/or a passive strategy that gets access to different % of the pool-cost? It would be very interesting to see how this would compare in Figure 3d.

  5. Why consider a pool of experiments to "buy" and not allow querying the full range of scale and hyperparameters? Do you have enough data to construct a surrogate benchmark instead of a tabular one?

  6. This is contemporaneous work, but the authors may want to discuss how their problem setting differs from the loss-extrapolation setting in "ScAn-Bench: Evaluating Scaling Analysis Methodology (NeurIPS 2026, https://arxiv.org/abs/2609.35707)". Why do the baselines (non-contemporaneous) used there not also apply here?

2 comments

We thank the reviewer for recognizing the importance and timeliness of our work, and for the thoughtful comments. We address the points in turn.

As benchmark papers set the standard for evaluation protocols, the experiments for this paper need to be especially well designed. However, there are several crucial flaws. Experiments are run with three seeds only and the error in the measurement is not quantified in the main results (e.g., Table 1), the authors themselves mention an alarming difference across repeats ("Outcomes vary across repeats." paragraph). Also the random baselines are omitted in the main results.

We fully agree that the experiments of a benchmark paper need to be carefully designed and their results carefully interpreted. We explain below.

The conclusions of the main results hold despite the variation across repeats. Single runs do vary considerably, which is what the "Outcomes vary across repeats." paragraph describes; but Table 1 reports, for each system, the median over 30 tasks × 3 repeats, i.e., 90 runs. We bootstrap the three repeats within each task (100,000 resamples) to obtain a 95% interval for each system's median R² (Table R1).

Table R1. Median R² of each system with its 95% bootstrap interval

SystemPassiveActive
Kimi K30.742 (0.729–0.763)0.555 (0.405–0.630)
DeepSeek V4 Pro0.728 (0.718–0.739)0.540 (0.476–0.639)
GPT-5.6 Sol0.728 (0.703–0.749)0.514 (0.404–0.573)
Qwen3.8 Flash0.746 (0.731–0.771)0.398 (0.191–0.468)
Hy4-Preview0.737 (0.725–0.760)0.350 (0.234–0.483)
Gemini 3.8 Flash0.753 (0.750–0.765)0.255 (0.186–0.444)
DeepSeek V4 Flash0.723 (0.696–0.735)0.235 (0.138–0.405)

For all seven systems, the Active interval lies entirely below the Passive interval. The Passive intervals are only about 0.02–0.05 wide, while in Active the intervals of the strongest and weakest systems do not overlap (e.g., DeepSeek V4 Pro 0.476–0.639 vs. DeepSeek V4 Flash 0.138–0.405). Hence neither core conclusion, that Active scores fall below Passive and that Active separates systems, is an artifact of noise across repeats.

The random baseline is reported in Appendix Table 9 (Table 6 in the submission): buying experiments at random and fitting them with the reference fitting procedure gives a median R² of 0.150 at 20% of the cost and 0.360 at 50%, whereas the three strongest systems reach 0.514–0.555 at median costs of 18–35%, above the random baseline at every budget.

We thank the reviewer for this reminder, and have added these intervals and the random baseline to Table 1 (Appendix B.2).

Q1.

Why categorize scenarios as eligible or not in terms of difficulty? Would it not be interesting to consider the scenarios with R2 > 0.8?

Easier tasks cannot distinguish methods. On tasks with R² > 0.8, a simple law already scores highly, leaving little need to discover a law and little room to separate methods. We therefore keep only tasks on which the reference fitting procedure (Appendix A.2) scores R² of at most 0.8 in both settings. Table R2 gives three examples; for instance, training step alone predicts IMP-ACT return with R² of 0.9273.

Table R2. Three tasks on which a simple law already fits well

TargetLaw fitted per groupInputsTask R²
Language-model lossE+A/(1+D)pE+A/(1+D)^pD is the number of examples seen. Model size is fixed within each group.0.9755
IMP-ACT returnE+A/(1+t)pE+A/(1+t)^pt is the training step0.9273
LAION segmentation errora+bln⁡(1+N)+cln⁡(1+C)a+b\ln(1+N)+c\ln(1+C)N is parameter count in millions. C is training FLOPs.0.9647

That said, we agree that these tasks are also of interest. We have added these examples to Appendix A.2 (Table 5) and will release such tasks for comparing the cost of reaching a given accuracy.

Q2.

Why remove uninformative features at the level of the benchmark design? Is it to help the search method?

This is data cleaning applied identically to all methods, and it favors none. We remove only input columns that carry no information for any method, such as constant columns, and every system receives the same inputs (Appendix A.1). For example, IBM BoolQ lists price as 6ND; keeping parameter count N and training tokens D still allows a law to model their product or their separate effects.

Q3.

Why would a method like CARBS (Fetterman et al.) not be applicable to your setting? Is it only because it is supplied a cost-cap? If so, why not benchmark these kind of methods for a couple of pre-set costs? May that be a strong baseline?

CARBS covers only one step of Active SLD: choosing experiments for a given law under a preset cost. In Active SLD, the agent must also propose and revise the law and decide how much to spend. Nevertheless, we evaluated CARBS at two cost levels taken from the agents' own runs. At each cap we mapped CARBS proposals to the nearest affordable unbought candidate and refitted the agent's own law on both purchase sets.

Table R3. Median test R² of the same law at matched cost caps

Cost cap (median)System (tasks)Agent's pointsCARBS points
First purchases (0.5–0.9%)DeepSeek V4 Pro (30)−0.431−1.048
First purchases (0.5–0.9%)Kimi K3 (27)−0.574−1.591
Whole run (69–79%)DeepSeek V4 Pro (12)0.5600.408
Whole run (69–79%)Kimi K3 (11)0.5630.561

At both cost levels, CARBS does not outperform the agents' own selections, and it is clearly worse at low cost. CARBS is therefore not a stronger baseline on these tasks, and because it needs the law and the cost to be given, it cannot by itself perform Active SLD. We have added this comparison to Table 2 (Section 3.4) and Appendix B.4.

Q4.

Why not compare to a referemce law and/or a passive strategy that gets access to different % of the pool-cost? It would be very interesting to see how this would compare in Figure 3d.

In Figure 3d, the random baseline at a 20% budget lies at about 20% cost and a 12.2-percentage-point rise in failure share. This is consistent with the trend that systems spending less fail more often, and shows that how much to buy is a key decision in Active SLD, one the agent must plan itself.

Table R4. Random purchases at fixed budgets

Budget (% of total price)Median test R² (Table 9)Failures in 90 fits
5%−0.00345
10%0.08540
20%0.15032
50%0.36028

A fit fails if R² is zero or negative or no score is returned.

We thank the reviewer for the interest and the reminder, and have added this baseline to Figure 3d and its failure counts to Table 10.

Q5.

Why consider a pool of experiments to "buy" and not allow querying the full range of scale and hyperparameters? Do you have enough data to construct a surrogate benchmark instead of a tabular one?

The pool design lets us evaluate how an agent plans large-scale experiments while ensuring that the outcomes it obtains are real. Answering an arbitrary query would require either running a new training job, which is infeasible at these scales, or predicting its outcome with a surrogate model.

Part of our data is sparse; for example, T29 has only 40 measured experiments in five groups. A surrogate fitted to such data would be unreliable: at unmeasured configurations it only provides another prediction, and one optimistic prediction could lead an agent to change its law or stop early. Using it as ground truth would test whether agents recover the surrogate rather than the true law.

We nevertheless think this is a good suggestion and will carefully consider its applicability. We thank the reviewer for it.

Q6.

This is contemporaneous work, but the authors may want to discuss how their problem setting differs from the loss-extrapolation setting in "ScAn-Bench: Evaluating Scaling Analysis Methodology (NeurIPS 2026, https://arxiv.org/abs/2609.35707)". Why do the baselines (non-contemporaneous) used there not also apply here?

The two settings differ in who makes the decisions. In ScAn-Bench, the budget and the extrapolation method are fixed in advance, and the task is to extrapolate optimal loss from data acquired under that budget. In Active SLD, the agent itself decides which experiments to buy, which law to build, and when to stop, with no budget cap. After a surprising result it can revise its law, buy an experiment that tests the new law, or stop, so experimental design and law discovery interact within one run. We also score every hidden configuration, for targets beyond loss such as accuracy and segmentation error.

Hence the ScAn-Bench baselines, random selection and CARBS (Section 4), cover only the purchasing step. We report both: the strongest agents stay above random selection at every budget (Table 9), and CARBS does not outperform the agents' own selections (Table R3). Neither can choose the law or the total spend. Active SLD thus evaluates how well an agent plans its own experiments, which the ScAn-Bench setting does not measure.

We have added this discussion of ScAn-Bench to the related work (Section 4).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Updates and Response to Reviewers

Dear Reviewers and Area Chair,

We thank the reviewers for their careful reading and constructive comments. We are glad that Reviewer R7LC sees the paper as targeting "a very important and timely problem" and expects "several future works to use this benchmark", that Reviewer uQcp finds the Active setting "a good idea" that "is implemented carefully", and that Reviewer YdfC calls it "a well-motivated setting for AI4AI research". Below we summarize our core contributions and the key updates in the revision.

Core Contributions

  • Active SLD benchmark and setting: We propose SLDBench v2 and active scaling law discovery (Active SLD), a new AI-for-AI setting in which agents must plan their own experiments and discover the law from what they buy, just as researchers must plan compute before training a large model. Agents buy recorded training runs at their real cost, a proxy that makes large-scale experimental planning feasible to evaluate.
  • Greatly expanded scale and coverage of SLDBench: We expand SLDBench nearly 16-fold, to 127 scenarios with 3.4M records across language, vision–language, and protein models, and select 30 hard tasks on which a standard power-law fit extrapolates poorly (R² ≤ 0.8); their source training runs total over 10²³ FLOPs.
  • Large-scale experimental planning is the key open challenge: Evaluating seven frontier agent systems, we find that their failures stem mainly from buying cheap small-scale experiments and stopping too early. Large-scale experimental planning is therefore the key open challenge for agents that assist model development, and a concrete target for improving them.

Key Rebuttal Updates

  • More complete results with uncertainty intervals and a random baseline (Table 1, Figure 3d): Addressing Reviewers R7LC and YdfC, every system's Active interval lies below its Passive interval, and our findings hold in 97.1–99.9% of source-level resamples. The new random baseline fails more often the less it buys, the same pattern as the agents.
  • New comparison with specialized methods (Table 2, Section 3.4): Addressing Reviewers YdfC and R7LC, SLDAgent matches the strongest agents in Passive (0.773) but cannot choose experiments, while CARBS and SL² do not select better experiments than the agents at the same cost. Each makes only one of the two decisions that Active SLD requires.
  • Improved analysis of the cost instruction and stopping decisions (Section 3.2, Appendices A.5 and C.1): Addressing Reviewers uQcp and YdfC, we give the full instruction and show that runs spending under 1% of the pool fail more than half the time; most sampled low-cost failures that give a reason judged their data or fit sufficient rather than citing cost.
  • Improved details throughout the paper: Addressing Reviewers uQcp, R7LC, and YdfC, we state the separation of systems more precisely, report the unscored runs, clarify clipping, pricing, and contamination, discuss the concurrent ScAn-Bench, and will release all 127 scenarios with all agent trajectories.

We believe these updates significantly strengthen the paper, and we hope they address the remaining concerns. We kindly ask the reviewers to consider raising their scores in light of these improvements.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.