- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies additive manufacturing with separate budgets for printing and destructive testing. SCOUT uses a residual Gaussian process to combine simulator predictions with design and infrared features. It selects which parts to print and then uses the observed infrared features to decide which parts to test. Experiments use a synthetic AM generator and public battery and concrete datasets. The main concerns are the acquisition function’s added value, the treatment of model learning across the printing and VoT testing stages, and the evidence for retaining the simulator and adapting to test scarcity.
Critical strengths
1. The paper defines a useful experimental decision problem. Separating printing from testing allows the method to use information collected during manufacturing before spending a test. The paper clearly states the two budgets, the order in which observations become available, and the requirement to accept or reject a test before the next print. It also explains how these constraints differ from those in function-network and partial-evaluation BO. (Sections 2–3, pp. 3–4.)
2. The model handles intermediate observations explicitly. For unprinted candidates, SCOUT marginalizes over unknown IR features. After printing, it uses the measured features to update predictions. The ablations and unobserved-state sweep show that IR helps within the generator and that its benefit decreases as more of the performance variation becomes invisible to IR. These results support using process observations in this setting. (Section 4.2, p. 4; Section 5.2, p. 7; Appendix F.6, pp. 26–27.)
3. The analysis explains a specific failure of the testing threshold. Under calibrated predictions and a continuous score distribution, the own-prediction threshold tests different designs at the same rate. The authors explain why this can reject a good part, propose a replacement, and trace the battery results to specific samples. Reporting results without those samples and without labeled history helps identify when the correction is useful. (Sections 5.3–5.4, p. 8; Appendix N, pp. 43–44.)
Critical weaknesses
1. The added value of the four-term acquisition function is not established. SCOUT combines performance, uncertainty, calibration value, and IR information value using largely heuristic weights. The reported 43% gain on NAB changes both printing and testing rules. It supports the full system but does not isolate the printing acquisition's contribution. With the same surrogate and testing rule, the battery experiments do not show a significant advantage over EI. A controlled comparison on NAB is needed to establish what the additional acquisition terms contribute. (Sections 4.3 and 5.1, pp. 5-7; Appendix B, p. 20; Appendix O, pp. 46-47.)
2. Printing and VoT testing do not consistently account for model learning. The outer CalVal term values a candidate for the model calibration that a complete test could provide. The inner VoT rule freezes the model and excludes the test's benefit for future learning. Calibration value can therefore favor printing a part without entering the subsequent decision to test it. This omission directly concerns the stated motivation of using scarce tests to correct the simulator. Its effect on test selection and final performance remains untested when labeled data are scarce and the model is inaccurate. (Section 4.3, pp. 4-5; Appendix N, pp. 43-44.)
3. Two central motivations are not sufficiently tested. The IR ablations support the value of process observations, but they do not establish whether retaining a biased simulator improves on directly modeling the objective with the same inputs and decision policies. Appendix F.5 acknowledges this gap. The main budget experiments also keep the testing-to-printing ratio at 0.6 while varying the total budget. They therefore do not establish how the method adapts as testing becomes scarce relative to printing. Both questions can be tested within the existing generator. (Section 5.1, pp. 6-7; Appendix F.5, pp. 25-26.)
Questions for authors
- On the NAB generator, what does the four-term acquisition gain over EI with the same surrogate, initial data, budgets, and testing rule? (Section 5.1, pp. 6-7; Appendix O, pp. 46-47.)
- How does VoT account for candidates favored by CalVal for model calibration? What is the effect of excluding future learning on test selection and final performance when labeled data are scarce and the model is inaccurate? (Section 4.3, pp. 4-5; Appendix N, pp. 43-44.)
- With inputs and decision policies held fixed, does retaining the simulator outperform directly modeling the objective? At a fixed printing budget, how does performance change as the testing budget decreases? (Section 5.1, pp. 6–7; Appendix F.5, pp. 25–26.)