acceptodds
Under review as a conference paper at ICLR 2027

SCOUT: What to Print and What to Test in Two-Budget Closed-Loop Design

Abstract

Materials design under additive manufacturing (AM) couples three oracles whose costs differ by orders of magnitude: a structurally biased finite-element simulator, an in-situ infrared (IR) video that every print records at no marginal cost, and a destructive mechanical test that only a few printed parts can receive. We formalize this as two-budget closed-loop design with a free intermediate observation, a relative of Bayesian optimization of function networks in which the intermediate observation comes only with a print and each test consumes a printed part from its own budget. We introduce SCOUT, whose residual Gaussian process corrects the simulator from design and IR features and whose two-stage acquisition chooses what to print and then, once the IR is observed, what to test. On a literature-anchored Nickel Aluminum Bronze generator, SCOUT attains lower simple regret than thirteen baselines, including design-of-experiments practice, at every budget, and at the smallest budget 43% lower regret than EPIG on the identical IR-equipped surrogate, 37% with IR noise raised to the dispersion of a real recording, a margin that narrows as the test budget grows. We show that a budget-fraction test threshold read against a part's own prediction tests every design at the same rate under a calibrated model and can discard the best part, and we derive a value-of-testing rule from the two budgets. In a pre-registered study on 150 random instances of public battery and concrete data, the new rule keeps the best battery cell that the old threshold discards in 39 of 53 runs with labelled history and lowers SCOUT's regret there by 8.6 and 12.8 points after Holm's correction, to the lowest mean of 21 arms, tied in one family. A post hoc analysis traces the effect to five cells whose recorded life merges two test batches, and without history a Gaussian process on the design alone is better. A physical AM campaign is left to future work.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Bayesian optimization, Bayesian experimental design, active learning, multi-fidelity optimization, Gaussian processes, function networks, closed-loop experimentation, additive manufacturing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WcggOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies additive manufacturing with separate budgets for printing and destructive testing. SCOUT uses a residual Gaussian process to combine simulator predictions with design and infrared features. It selects which parts to print and then uses the observed infrared features to decide which parts to test. Experiments use a synthetic AM generator and public battery and concrete datasets. The main concerns are the acquisition function’s added value, the treatment of model learning across the printing and VoT testing stages, and the evidence for retaining the simulator and adapting to test scarcity.

Critical strengths

1. The paper defines a useful experimental decision problem. Separating printing from testing allows the method to use information collected during manufacturing before spending a test. The paper clearly states the two budgets, the order in which observations become available, and the requirement to accept or reject a test before the next print. It also explains how these constraints differ from those in function-network and partial-evaluation BO. (Sections 2–3, pp. 3–4.)

2. The model handles intermediate observations explicitly. For unprinted candidates, SCOUT marginalizes over unknown IR features. After printing, it uses the measured features to update predictions. The ablations and unobserved-state sweep show that IR helps within the generator and that its benefit decreases as more of the performance variation becomes invisible to IR. These results support using process observations in this setting. (Section 4.2, p. 4; Section 5.2, p. 7; Appendix F.6, pp. 26–27.)

3. The analysis explains a specific failure of the testing threshold. Under calibrated predictions and a continuous score distribution, the own-prediction threshold tests different designs at the same rate. The authors explain why this can reject a good part, propose a replacement, and trace the battery results to specific samples. Reporting results without those samples and without labeled history helps identify when the correction is useful. (Sections 5.3–5.4, p. 8; Appendix N, pp. 43–44.)

Critical weaknesses

1. The added value of the four-term acquisition function is not established. SCOUT combines performance, uncertainty, calibration value, and IR information value using largely heuristic weights. The reported 43% gain on NAB changes both printing and testing rules. It supports the full system but does not isolate the printing acquisition's contribution. With the same surrogate and testing rule, the battery experiments do not show a significant advantage over EI. A controlled comparison on NAB is needed to establish what the additional acquisition terms contribute. (Sections 4.3 and 5.1, pp. 5-7; Appendix B, p. 20; Appendix O, pp. 46-47.)

2. Printing and VoT testing do not consistently account for model learning. The outer CalVal term values a candidate for the model calibration that a complete test could provide. The inner VoT rule freezes the model and excludes the test's benefit for future learning. Calibration value can therefore favor printing a part without entering the subsequent decision to test it. This omission directly concerns the stated motivation of using scarce tests to correct the simulator. Its effect on test selection and final performance remains untested when labeled data are scarce and the model is inaccurate. (Section 4.3, pp. 4-5; Appendix N, pp. 43-44.)

3. Two central motivations are not sufficiently tested. The IR ablations support the value of process observations, but they do not establish whether retaining a biased simulator improves on directly modeling the objective with the same inputs and decision policies. Appendix F.5 acknowledges this gap. The main budget experiments also keep the testing-to-printing ratio at 0.6 while varying the total budget. They therefore do not establish how the method adapts as testing becomes scarce relative to printing. Both questions can be tested within the existing generator. (Section 5.1, pp. 6-7; Appendix F.5, pp. 25-26.)

Questions for authors
  1. On the NAB generator, what does the four-term acquisition gain over EI with the same surrogate, initial data, budgets, and testing rule? (Section 5.1, pp. 6-7; Appendix O, pp. 46-47.)
  2. How does VoT account for candidates favored by CalVal for model calibration? What is the effect of excluding future learning on test selection and final performance when labeled data are scarce and the model is inaccurate? (Section 4.3, pp. 4-5; Appendix N, pp. 43-44.)
  3. With inputs and decision policies held fixed, does retaining the simulator outperform directly modeling the objective? At a fixed printing budget, how does performance change as the testing budget decreases? (Section 5.1, pp. 6–7; Appendix F.5, pp. 25–26.)
Reviewer_qxCVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper tries to address how to run a closed-loop design campaign in metal additive manufacturing (AM) when printing a part and destructively testing it have separate budgets, and each print also gives a free in-situ infrared (IR) observation. The authors formalize this as two-budget closed-loop design and propose SCOUT: a Gaussian process that corrects a finite-element simulator using design, simulator and IR features, a weighted acquisition score to choose what to print, and a second decision on whether to test each printed part. They also show that a budget-fraction test threshold read against a part's own prediction tests every part at the same rate under a calibrated model, and derive a value-of-testing rule instead. On a synthetic Nickel Aluminum Bronze generator, SCOUT has 43% lower regret than EPIG with the same IR-equipped surrogate; in a pre-registered study on public battery and concrete data, the new test rule improves regret in the battery families with labelled history.

Critical strengths

I think the work is done carefully, and the problem comes from a real need: in materials and manufacturing, printing, in-situ monitoring and destructive testing differ in cost by orders of magnitude, and modelling them with separate budgets is a reasonable abstraction (pp. 1-4). The paper also has one clear observation: if each printed part is compared against its own predictive distribution, the test rule rewards lucky parts and can discard the best one, so parts should be compared against a common threshold (Proposition 4 and Fig. 1b, pp. 2, 43). Figure 1 explains this well. I also appreciate that the authors disclose their limitations and post hoc changes in detail (pp. 8-9, 46-47).

Critical weaknesses
  1. I feel the novelty and the insight are far below the ICLR bar. The method combines known components with hand-set choices: a Kennedy-O'Hagan style residual GP on the simulator, a print score that is a weighted sum of four terms, and a quantile threshold for testing. The seven weights are "a heuristic motivated by a variance argument, not a derived optimum" (p. 20), uniform weights cost only 1.5 points of regret (p. 5), and the structured additive kernel differs from a plain concatenated kernel by 0.1 points, within seed noise (p. 4). The theory applies the GP-UCB bound with a slack term, does not cover the test decision, and "is not numerically informative at our budgets" (p. 5). The one real insight, Proposition 4, is a direct use of the uniformity of the probability integral transform (p. 43), and the value-of-testing rule built on it is not even used in the headline experiment: it "run[s] only in the open harness, not on the generator" (p. 9), so the 43% result uses the threshold the paper itself shows to be flawed. There is no learning or representation contribution.

  2. I think the evidence does not support the story the paper tells. The title and framing are about AM, but all AM results come from the authors' own synthetic generator, which is "not evidence from a physical machine" (p. 6). That generator emits the IR features as a noisy measurement of the latent that drives the target, "a link that §4.1 does not support for real NAB" (p. 6), and none of the thirteen generator baselines use IR (p. 7), so the main win is largely built into the setup. On real data the picture is weak. On battery data, the effect of the new test rule comes from five cells whose recorded life merges two test batches (p. 8); without them the difference is -0.2 and +1.5 points, p = 0.95 and 0.53 (p. 49). On concrete, the rule is not significant after Holm correction (p = 1.00 and 0.10, Table 20), the data "do not separate SCOUT's print rule from EI's" (p. 8), and on function networks SCOUT "does not separate from most IR-equipped arms" (p. 9). In my view this work would fit an applied materials or manufacturing venue better than ICLR.

Other weaknesses:

  • The paper is very hard to read. A simple question is spread over 53 pages with long, number-heavy sentences and many acronyms; one sentence in the abstract carries six numbers.
  • The Holm split into battery and concrete families "was chosen after the results of two battery families had been seen" (p. 46), and several arms and settings were "added after the review" (pp. 9, 47), which weakens the pre-registration claim.
  • The IR features measure the 565°C isotherm, which, as the authors note, is "not a κ-phase fraction" (p. 4), so the physical link between the IR record and the property of interest is not established.
Questions for authors
  1. Are there any real paired (IR, mechanical property) measurements that test whether the IR record explains the simulator's error? Without them, how should the AM results be read?
  2. Without the five merged battery cells, what evidence remains that the value-of-testing rule improves on the own-prediction threshold?
  3. (Clarification) Why does the headline generator experiment use the own-prediction threshold rather than the value-of-testing rule, and how does SCOUT with the value-of-testing rule perform on the generator?
Reviewer_LhjmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies closed-loop design with separate manufacturing and testing budgets, where manufacturing reveals an intermediate observation before an expensive test. SCOUT combines a simulator-based residual Gaussian process with a two-stage acquisition policy. The paper also analyzes a failure mode of prediction-relative testing thresholds and proposes a value-of-testing rule. Evaluation uses a synthetic additive-manufacturing benchmark and retrospective battery and concrete datasets, demonstrating benefits under some settings but limited evidence of consistent advantages over simpler alternatives.

Critical strengths
  1. Relevant problem formulation: Separate manufacturing and testing budgets reflect a practical constraint in experimental design.
  2. Informative threshold analysis: The analysis identifies a failure mode that can affect how intermediate observations guide testing decisions.
  3. Acknowledgment of limitations: The paper reports negative results and examines factors that limit the interpretation of its empirical gains.
Critical weaknesses
  1. Uncertain incremental benefit: The current comparisons do not clearly establish the acquisition policy's advantage over simpler methods using the same information and testing rules.
  2. Limited evidence of generalizability: The complete framework is evaluated on synthetic data, while the main testing-rule gains on real data appear concentrated in a small subset of samples.
  3. Scope of the claims: Some aspects of the framing could better reflect the heuristic nature of the acquisition design and the limited scope of the theoretical and empirical support.
Questions for authors
  1. Matched comparisons: Could the authors clarify whether SCOUT retains an advantage when competing policies use the same surrogate, observations, and testing rule?
  2. Algorithm specification: Could the authors specify which testing threshold was used in the main synthetic experiments and explain its relationship to the proposed value-of-testing rule?
  3. Robustness of the gains: Could the authors clarify what evidence supports the testing rule's effectiveness beyond the small subset of battery samples driving the reported improvement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.