- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces construction laws that feature a performance target, resource budget, and an identified mechanism and maps that to a recipe, forecast, and qualification test. Unlike scaling laws that predict performance along a chosen path, construction laws actually decide which system needs to be built for a desired capability. These laws are developed for resource allocation, numerical precision, reusable information, and conditional specialization. Experiments show storage savings from numerical-precision training recipes and reduced task-local queries through reusable structure. However, the prediction of 2x query reduction in the HotpotQA setting does not hold on validation.
Critical strengths
- The distinction between scaling laws and construction laws is interesting. In particular, the two-resource example clearly shows that even exact knowledge of performance along one scaling path may not identify the optimal resource allocation.
- The identify/bound/select/freeze/qualify steps from Table 1 is a useful research methodology. I also appreciate that the paper honestly reports a negative result with the 2x budget crossing forecast.
Critical weaknesses
- The superintelligence framing is much broader than the experiments. In the paper, it is defined as “reliable performance beyond a declared expert reference over a declared breadth of tasks,” but the experiments study numerical precision on TinyStories, reuse in a synthetic setting, and conditional shard retrieval. These experiments show specific examples of construction laws but do not validate the broader superintelligence framing.
- It is unclear whether the construction-law procedure selects better recipes than alternative model-selection approaches. The numerical-precision experiment does not compare against a matched-budget human-tuned or Bayesian search, does not test the neighboring five-bit recipe for E2, and does not establish minimum-storage optimality. Therefore, it is unclear how much benefit comes from the construction-law procedure rather than the fitted response model itself.
Questions for authors
Clarification Requests:
-
For Theorem 3.1, Appendix K says that the mathematical tools are established and the contribution is in combining them into target-to-recipe statements. It is still unclear to me which part of the guarantee is technically new rather than an application of existing robust optimization results.
-
How sensitive are the selected numerical-precision recipes to uncertainty in the fitted response model? The precision coefficient reaches its imposed upper bound, and the mechanism checks have a maximum residual of nats/byte. Do alternative reasonable fits select the same models and bit widths?
Suggestions for Future Work:
- Can the authors evaluate the neighboring five-bit recipe for E2 or compare against a matched-budget model-selection baseline? This would help clarify whether the construction-law procedure selects a better recipe or only a viable one.
3 comments
Thank you again for suggesting the direct test of the E2 five-bit neighbor. We have now completed the experiment, and the result is informative for both the empirical claim and the construction procedure.
We trained architecture C (7.123 million parameters) at five bits, preserving the original 70,189,056 processed training tokens per seed, the training-work cap, and both seeds.
The five-bit model satisfies the 0.010-nat comparative noninferiority criterion against the eight-bit reference in both seeds.
On the test stream, the paired five-bit minus eight-bit loss differences are +0.004390 and +0.003054 nats/byte. The Bonferroni-adjusted simultaneous 95% upper limits are 0.005124 and 0.003738, respectively, both below the declared 0.010 threshold.
The packed-weight storage measurements are:
- Eight bits: 7.169 MB
- Originally selected six bits: 5.391 MB
- Newly tested five bits: 4.501 MB
Thus, the five-bit construction achieves a 1.593× storage ratio relative to the eight-bit reference, compared with 1.330× for the originally selected six-bit construction. It uses 16.5% less storage than the six-bit recipe while satisfying the same empirical qualification criterion.
This finding also reveals a limitation in the original selection. The inverse selected six bits under a fitted absolute-loss target, while the empirical qualification rule permits a 0.010-nat degradation relative to the eight-bit comparator. The six-bit recipe qualified, but the newly tested five-bit neighbor also qualifies at lower storage cost.
We will revise the paper to report this result and align the discussion of optimality with the actual qualification criterion.
The new models were trained after your review using a fixed protocol. Their evaluation reuses the test stream from the original submission, so we report this as post-review evidence with evaluation-batch uncertainty conditional on the two seeds. An independent holdout, broader fit-sensitivity analysis, and matched-budget model-selection comparison would strengthen the conclusion further.
We appreciate the suggestion. It has produced a useful additional measurement and identified a concrete improvement to the construction methodology.
Official Response From Reviewer
Thank you for the response and for running the five-bit E2 experiment. The result confirms that the original procedure did not select the lower-storage qualifying recipe. Additionally, the fit-sensitivity and matched-budget comparison still remain unresolved. I therefore maintain my rating.
Thank you for the constructive review. The requests to test E2 at five bits and to assess fit sensitivity go directly to the strength of our empirical conclusions.
The precision experiment selected recipes using a fitted absolute-loss target and then qualified them using a predeclared 0.010-nat comparative noninferiority rule. All six tested target–seed comparisons satisfy that rule, with packed-weight storage reductions of 1.33 to 1.59 times relative to their references. This establishes prospective qualification for those constructions. Minimum-storage optimality requires additional comparisons.
In particular, the untested E2 five-bit neighbor has a calculated packed-weight size of 4.501 MB, compared with 5.391 MB for the selected six-bit recipe. The original response model predicts a loss of 0.820668 for this five-bit configuration, above the nominal absolute target of 0.818168. Its performance under the separate comparative criterion remains an empirical question.
We have prepared a controlled two-seed five-bit E2 experiment with the original architecture, token count, training-work cap, data, and optimizer. Only precision changes. We will report the observed result, including a failure if the candidate does not qualify.
The fitted precision coefficient also reaches its imposed upper bound, and the largest held-out mechanism residual is approximately 0.0401 nats per byte. We are examining how reasonable alternative fits affect the selected recipes. We will keep the general uncertainty-qualified theorem separate from experimental claims supported by this particular fitted response.
Your suggestions identify exactly which evidence can establish the reliability and limits of the selected recipes. We appreciate them.