- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces ArchitectureIQ, a benchmark of 500 three-choice questions. Each question shows a synthetic dataset (regression, tabular/XOR/spiral classification, or bigram LM) and three training recipes (model plus optimizer settings, sharing the same data, loss, batch size, and step budget). The test-taker predicts which recipe reaches the lowest final test loss. Ground truth comes from running each recipe for 10 seeds, and a question is kept only if the winner's worst seed beats every rival's best seed.
Frontier LLMs reach about 76% accuracy (85–87% on optimizer-only questions, 38–61% on architecture-only). Ten human ML researchers average 51% on a 50-question subset, and the best scores 66%. The paper then argues that LLM "model intuition":
- is unstructured, because few-shot examples and more reasoning effort don't help;
- is compressible, because a 20-proposition knowledge base distilled from Claude Opus 5 lifts GPT-4o from 53% to 68% on a 50-question subset;
- is insensitive to data, because on 52 paired questions where only the dataset changes and the winner flips, accuracy drops to 46–50%.
Critical strengths
- The execution-grounded design is solid. Every label comes from actually running the generated code over 10 seeds with a strict separation filter, choice order is randomized, and the release audit checks prompt–program agreement and leakage (Appendix A.3–A.4).
- Using synthetic datasets to avoid contamination from known benchmarks is a sensible choice. The generator allows controlled variants, such as varying the number of choices (Fig. 3) and paired dataset swaps (Sec. 3.2, Appendix E).
- The paired label-flip evaluation is the most interesting result. Models switch their answer on only 6–23% of pairs when only the dataset changes and the correct answer flips. This is a clean, informative test of whether predictions use the data at all.
- The paper compares against non-LLM baselines (heuristics, logistic regression, a depth-4 tree, and an analytical decision policy), which helps interpret what the LLM numbers mean.
Critical weaknesses
-
The benchmark mostly measures short-horizon optimization speed, not "model intuition" in the sense claimed.
- All models are tiny: MLPs of width ≤256 and depth ≤5, or 1–4 layer Transformers/GRUs.
- Datasets have 256–2,048 points, and training runs only 256–2,048 steps (Appendix A.2).
- In this regime, the winner is largely determined by how far parameters move in the given steps. The paper's own evidence confirms this:
- an analytical policy based on an integrated effective-learning-rate proxy reaches 78.2%, above every LLM (Appendix C.3);
- logistic regression on configuration features reaches 74.2%;
- most knowledge-base entries are budget-specific rules such as "Within T ≲ 256 steps, rank candidates by stable effective step size" (Fig. 6).
This explains why optimizer-only questions are easy (85–87%) and architecture-only questions are near chance. Architectural knowledge from real-world practice (depth, residuals, LayerNorm help) often doesn't apply to 256-step training of tiny MLPs. Table 4 supports this reading: on architecture-only questions, GPT-4o (52%) and Llama 3.1 70B (49%) outperform most frontier models (38–48%). That suggests frontier models apply real-world priors that are wrong in this regime, not that they lack intuition. The title and abstract ("do language models have as good intuition about model training as top AI researchers?") claim far more than this regime supports.
-
The "first benchmark" claim is incorrect, and closely related work is missing. "Predicting Empirical AI Research Outcomes with Language Models" (Wen et al., NeurIPS 2025) already benchmarks LLMs and 25 human experts on predicting which of two research ideas performs better on real benchmarks, with execution-grounded labels from papers. LLM-assisted hyperparameter optimization (e.g., LLAMBO, ICLR 2024, and LLM-as-HPO work) also directly studies whether LLMs can rank training configurations before running them. These need to be discussed. The contribution should be repositioned as a controlled, synthetic, executable variant of this problem.
-
The human comparison is too small to support the stated conclusions.
- Humans answered a different 50-question subset (22 architecture, 16 optimizer, 12 mixed), while models are scored on all 500.
- The headline compares the best of 10 humans with models, which adds a selection effect.
- The claim "LLMs are sub-human on architecture" (GPT-6 Astra 38% vs. best human 64%) rests on 22 human items. A 95% CI on 22 items is about ±20 points, and the human mean is 51% overall.
- The abstract also says humans "understand too little about data," but humans were not evaluated on the paired data-flip questions at all.
- The human setup is not described: time limits, whether humans could run code, and what instructions they received.
All comparisons should be on the same items, report mean human performance with CIs, and avoid human claims that have no data behind them.
Questions for authors
- What fraction of sampled questions survives the 10-seed separation filter, and how does the retained distribution differ from the unfiltered one (e.g., in the learning-rate ratio between candidates)?
- How do model rankings and the architecture-vs-optimizer gap change with longer budgets (e.g., 10× steps) or tuned learning rates per candidate?
- Why do GPT-4o and Llama 3.1 70B beat most frontier models on architecture-only questions? Do frontier models systematically prefer deeper, residual, or LayerNorm candidates that lose at short budgets?
- Can you report human mean accuracy with CIs on the same items as the models, and describe the human protocol (time, tools, instructions)? Were humans evaluated on the paired data-flip questions?
2 comments
Thank you for the detailed response and the additional experiments.
I am satisfied with the plan to discuss Wen et al. and LLAMBO and to reposition the contribution; please also revise the "first benchmark" statement in the contribution list. The same-item human comparison, the CI, and the protocol description address most of my concerns about the human study. I also accept the Question 3 evidence: models select maximum depth and LayerNorm less often than the winners do, so I withdraw my suggestion that frontier models lose because they over-apply depth/LayerNorm priors.
Weakness 1 remains open. My main concern was not that explanation, but that outcomes in this regime are largely predictable from an effective-step-size proxy. The response does not address the paper's own evidence for this: the analytical effective-learning-rate policy (78.2%) and logistic regression on configuration features (74.2%) match or exceed every LLM. Two new results in the rebuttal reinforce the concern:
- The filter shifts the lower quartile of the max/min learning-rate ratio from 1 to about 3.33, so retained questions are enriched for large learning-rate gaps.
- At 10x budget, accuracy drops by 7–23 points with everything else fixed (e.g., Claude Opus 5 from 86.7 to 63.3). This suggests predictions track short-horizon behaviour rather than a budget-aware understanding of training.
Given this, i will prefer to maintain the current score.
We thank the reviewer for the careful analysis and concrete suggestions.
Weakness 1: Does the benchmark mainly measure short-horizon optimization?
We do not fully agree that the scores are a consequnce of frontier models applying real-world priors that are inappropriate for short-horizon regime.
First, training budgets must be interpreted relative to the model and dataset sizes. Our recipes use 256-2,048 steps and 4,096-32,768 training samples seen. Many of the runs already converge within these budgets. Second, each question explicitly specifies the dataset, candidate recipes, optimizer, and budget. Deciding whether prior experience applies to these conditions is itself part of the capability being evaluated. Third, strong models do correctly predict many outcomes in this regime: Sol achieves 76.4% on Main500. The evidence in our response to Question 3 also does not support an explanation based on excessive preference for depth or LayerNorm.
Weakness 2: Novelty and missing related work.
We will discuss Wen et al. (NeurIPS 2025) and LLAMBO (ICLR 2024) and clearly state that our contribution is controlled, synthetic, executable training-outcome prediction.
Weakness 3 / Question 4: Human comparison, uncertainty, and protocol.
Figure 1 identifies the human evaluation as a fixed 50-question subset. On these same items (22 architecture, 16 optimizer, 12 mixed), the human mean is 51.0% (95% CI: 43.8%-58.2% across ten participants). The best score is 66.0%; Opus 5, Sol, and GPT-4o each score 64.0%, and V4 Pro scores 68.0%. As also explained in our response to Reviewer tGMv, we will revise the headline comparison to state that top models achieve scores comparable to the best human score on the same questions.
All ten participants have published at leading AI conferences. They predicted which recipe gives the best test metric, with no time limit, only pen and paper, and no internet, code execution, LLMs, or correctness feedback.
Question 1: Filter survival rate and distribution shift.
In a question-forming pipeline run, 204,064/464,944 (43.9%) candidate triples entering the ten-seed separation test survived. For the max/min learning-rate ratio, the median remains 10, while the lower quartile shifts from 1 to approximately 3.33 among passing triples. This changes the retained distribution, but does not invalidate the target capability: predicting execution-grounded outcomes for explicitly specified training recipes.
Question 2: Longer budgets and learning-rate tuning.
We evaluated four representative models on 60 fixed questions (20 per type) at 1x and 10x budget, keeping datasets, candidate recipes, learning rates, and seeds fixed:
Sol overtakes Opus at 10x budget; the other relative rankings remain unchanged. Optimizer accuracy remains higher than architecture accuracy for all four models, so the architecture gap persists with longer training.
Question 3: Do frontier models overprefer depth, residuals, or LayerNorm?
On pure-MLP questions where each attribute varies, all six examined models (Claude Opus 5, Claude Sonnet 5, Gemini 2.5 Pro, Gemini 3.1 Pro, GPT-4o, GPT-5.6 Luna) select maximum depth and LayerNorm less often than the winners do. Opus selects maximum depth on 52/124 items versus 78/124 actual winners, and LayerNorm on 46/68 versus 63/68. Some models show a slightly stronger preference for residual connections (Opus: 82/108 selections versus 67/108 winners). These descriptive patterns do not support a universal depth/LayerNorm explanation.