- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper builds on SLDBench, which asks LLM agents to discover a scaling law from given data that can predict the performance of larger runs. They add to this by collecting 127 scenarios (~3.4 million records) and selecting 30 tasks from 14 sources that satisfy R^2 <= 0.8 using a reference power law in both settings. They also add an Active setting where the agent is able to buy experiments from previously recorded runs (spending is recorded and signed, with a separate verifier) and is tested on the same hidden tests as the Passive setting. Cost is calculated by taking the cost of the experiments purchased divided by the cost of all available experiments and is reported but not used to determine the score. They find that all 7 systems (3 repeats) have medians in the Passive setting between 0.72 and 0.75 and in the Active setting between 0.24 and 0.56. All systems perform 1.5-4.7 times worse (MSE) when they have to purchase their data and systems that spent the least amount had the highest increase in failures. They also find that using a fixed scaling law and purchasing data that spans the prices of each group performs better than purchasing the cheapest experiments from each group.
Critical strengths
-
Adding an Active setting to SLDBench is a good idea and it is implemented carefully. By allowing agents to purchase previously run experiments, the authors are able to test if agents can effectively choose what data they want to collect to achieve their goal. By replaying previous runs they allow these types of questions to be tested more cheaply and reproducibly. Having a root-only oracle along with a signed ledger and a separate verifier allows the agent to be checked on its actions. Since both Active and Passive share the same testing data, the authors can directly compare how much better an agent is at discovering scaling laws without data collection. I believe with the addition of 127 scenarios this will be a helpful benchmark for future agents.
-
All systems perform worse when they have to purchase their data. Error increases ranged from 1.49x to 4.71x with all 95% confidence intervals above 1 in figure 3c. The authors also show that using a fixed scaling law and purchasing data that covers the prices of each group seems better than purchasing the cheapest points from each group.
Critical weaknesses
-
When discussing agent behavior they should mention that the agent is instructed to spend as few FLOPs as possible (Task T22 Prompt in Figure 2c). On line 177 we see the prompt that states this. Lines 214-215 show that cost is not taken into account when scoring the agent. The text only mentions that the prompt informs the agent that spending is being tracked on lines 205-207. If this is stated in all prompts then lines 21-22 and 311-317 should mention this when stating that agents like cheaper experiments and systems that spent less failed the most.
-
Systems are not clearly differentiated by the Active setting. Line 254 states that Active SLD is able to differentiate systems. Table 5 shows that systems all have similar percentages of Active runs with an R^2 greater than 0.5 (41-51%). Kimi K3 is also able to achieve the highest median because they are not including runs that don't have a score. There are 41 out of 90 runs that have an R^2 greater than 0.5, which with a median of 0.555 would mean at least 8 runs did not have a score. They already state on lines 362-363 that system medians are not a precise measure. So the statement that Active SLD is able to differentiate systems should be modified to reflect that. Also the reference law used to generate predictions clips its values to the min/max of the training target values (line 647). Since this is used to determine which tasks to use and in the non-agent baselines, they should mention this in the main text.
These are all things that could be easily addressed in the wording of the paper, so I lean towards acceptance.
Questions for authors
- Was "Spend as few FLOPs as you can." included in all of the task prompts? How do you think this affects the agents' spending?
- How many runs per system in the Active setting did not have a score, and why?
- Does the agent know what it's trying to extrapolate to or the input of the test set? If not, how do they know what experiments to buy? (line 355)
- Does the clipping mentioned on line 647 happen for each group or overall? How do you think this affects the task selection and section 3.4?
- Which tasks price checkpoints at the cumulative cost of their run (line 713)? What are the "task-specific columns" for Passive only? (line 708)
- Do you think agents may know scaling laws that have been published on these tasks? Will you be releasing the agent trajectories and all 127 scenarios?
2 comments
Q4.
Does the clipping mentioned on line 647 happen for each group or overall? How do you think this affects the task selection and section 3.4?
We reapplied the original selection criterion without clipping and selected exactly the same 30 tasks. We use this simple safeguard only in the reference law to limit extreme extrapolations. Within each task, all groups share one pair of bounds derived from the measurements used to fit the law.
Suppose the language models used to fit the reference law have validation losses ranging from 4 to 6. The difference is 2, so extending the range by 2 at each end gives bounds of 2 and 8. If the law predicts a loss of 1 for a larger model whose measured loss is 3, we raise the prediction to the lower bound of 2 and score it against 3.
For Section 3.4, we kept each strategy’s purchases and fitted law fixed and scored its predictions both ways. At all seven partial budgets, buying at different prices within each group gave higher median test R² than buying in ascending price order, with or without clipping. Table U3 shows the 2% budget. We have stated this rule in Section 2 and described it in full in Appendix A.2.
Table U3. Median test R² at a 2% budget cap
| Selection within each group | With clipping | Using raw predictions |
|---|---|---|
| Buy in ascending price order | −1.456 | −2.040 |
| Buy across the price range | −0.005 | 0.015 |
Q5.
Which tasks price checkpoints at the cumulative cost of their run (line 713)? What are the "task-specific columns" for Passive only? (line 708)
Tasks priced by cumulative training cost per checkpoint include T01 to T12 from ColPret, T16 from Gemstones, T23 on loss transfer, T24 from Misfitting, T27 from OpenWebText and T28 from ProteinLM. Translation task T29 uses 6 × parameters × cumulative examples seen. Two checkpoints from one run are charged separately. Revisiting the same checkpoint is free.
The extra Passive training fields vary by task. On T01 they are model name, training data, tokenizer, effective parameters, reported training tokens, source row and split. T22 includes the original mIoU, with 1 − mIoU as its target. Submitted laws receive the same test inputs and groups in both settings. We have added these details to Appendix A.3.
Q6.
Do you think agents may know scaling laws that have been published on these tasks? Will you be releasing the agent trajectories and all 127 scenarios?
During evaluation, agents have no network or web search, so they cannot look up source papers, and they never see the hidden test targets. Prior exposure during pretraining is still possible, since the source papers and their laws are public. As in SLDBench, we therefore treat SLDBench v2 as a literature-aware benchmark: like a human researcher building on prior work, an agent may draw on published laws, and the benchmark measures whether it can turn that knowledge into accurate extrapolation from the data it sees or buys. Recalling a published law also helps less here: the 30 tasks were selected because a standard power law extrapolates poorly (R² ≤ 0.8), and in the Active setting a known law form does not tell the agent which experiments to buy or when to stop.
We will release all 127 scenarios, including the 60 with priced experiment catalogues, and the full trajectories of all reported agent runs, including conversations, tool calls, purchases and costs where applicable, and the submitted laws. The revision adds this discussion to Appendix A.3 and the release plan to the reproducibility statement.
We thank the reviewer for recognizing the Active setting and its careful implementation, and for the thoughtful comments. We address the points in turn.
Active SLD still separates systems far more than Passive SLD, and lines 362–363 do not contradict this. In Passive, the seven medians span only 0.723–0.753; in Active, they span 0.235–0.555, and each system's error grows by a different factor, from 1.49× to 4.71×, with every 95% interval above one (Figure 3c), as the reviewer also notes. Although the share of Active runs above 0.5 is similar across systems (41–51%), the share of failed runs is not (21.1–40.0%), and for extrapolation it is precisely these failures that would mislead a decision about a larger run.
Lines 362–363 concern a finer question: the order of adjacent systems whose medians are close. Kimi K3's first place, which depends on how unscored runs are counted, is such a case. It does not change our conclusions.
That said, we fully agree that a claim about separating systems should be stated with care, so that it is not read as a precise ranking. We have therefore revised this statement (Section 3.2) and report the unscored runs and the resulting medians in Appendix B.1.
Q1.
Yes. All 30 Active task prompts include the sentence “Spend as few FLOPs as you can.” Removing this request in a reduced ablation raised median spending from 0.64% to 27.88% of catalogue cost for Qwen3.8 Flash and from 39.97% to 63.52% for DeepSeek V4 Flash.
Table U1. Median spending and failures with and without the cost instruction (six runs each; Appendix C.1)
We report R² and experiment cost separately. The revision states this instruction in Section 2, gives the full text in Appendix A.5, and discusses its effect in Section 3.2.
Q2.
Table U2 covers 90 scheduled Active runs per system. Nine Kimi K3 runs and three Hy4-Preview runs timed out. Every run of the other five systems returned a finite score. Table 1 takes the median of finite scores. Table 5 (Table 7 in the revision) uses all 90 runs as its denominator.
Table U2. Active runs without a finite score
Table 1 pools the scored runs from all three rounds before taking the median. By that measure Kimi ranked first at 0.555. Ranking unscored runs last yields 0.405, third behind DeepSeek V4 Pro and GPT-5.6 Sol. The seven shares above 0.5 are close, ranging from 41% to 51%.
Q3.
Agents know the target and extrapolation direction, but do not see the test configurations while buying. They see candidate inputs, groups and prices. T07 asks for normalized PIQA accuracy at higher token counts. An agent could buy two token scales for one model, observe how accuracy changes, then decide whether to buy a larger scale or probe another model group. The verifier supplies hidden inputs and group IDs to the submitted law only at scoring. We have added this to Appendix A.3.