- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents Caelum which constructs forward surrogates and differentiable inverse-design pipelines from CAE datasets. It collects preferences through controlled build sweeps, consolidates experience into explicit rules and retrieved precedents, and applies DPO to examples not covered by rules. The system is demonstrated across fifteen datasets in four physics domains, with evaluations of build decisions, forward predictions, and mostly surrogate-evaluated design optimization.
Critical strengths
This paper tries to address a practical CAE engineering bottleneck and implements the complete workflow across fifteen datasets. The controlled build sweeps reveal substantial changes in model rankings across data regimes (Sec. 4.4; Fig. 4). This empirical evidence has practical value for constructing surrogate pipelines and gives the work more substance than isolated agent demonstrations.
Critical weaknesses
There are two main cons, mainly regarding the evaluation and the claim:
First, the central learning contribution is not established by the evaluation. RRPO applies standard DPO to examples not covered by rules (Sec. 4.3), with limited methodological separation from existing rule-guided training, such as that of Zhao et al. (2026). More decisively, Appendix C states that all seven Board B decisions are resolved before the LLM tier, while precedent voting without an LLM already answers all fourteen statute-silent Board A items. Swapping between monolithic and residual adapters changes no deployed decisions (Table 9). Both boards also come from the same campaign used to construct the rules and precedents (Sec. 5). These results demonstrate recovery of accumulated build experience but do not establish transfer to unseen datasets or an additional benefit from RRPO.
The retention comparison likewise changes the routing architecture, comparing a bare adapter against the protected cascade (Fig. 6). A controlled evaluation on unseen tasks that actually require the learned tier is needed to substantiate the proposed learning contribution.
Second, the physical inverse-design capability is still insufficiently validated for me. Most improvements are evaluated by the same surrogate used to optimize the geometry (Sec. 5.2; Appendices D–E). Only two AF-Panel examples are checked using their source solver. For the cambered airfoil, a predicted 0.69% objective reduction becomes a 0.21% increase under solver evaluation, even though the design respects the displacement budget (Table 11). The discrepancy is small but demonstrates that bounded deformation alone does not validate the direction of improvement.
No corresponding solver verification is provided for the main three-dimensional design results. Producing improved physical designs is a central output of the proposed compiler, so this gap materially limits the claimed contribution.
Questions for authors
Were SuperWing, HiLift, and AeroCar evaluated with the rules, precedent library, and adapter frozen before any of their build outcomes entered the system? Please clarify the chronology and dataset-level separation.
On unseen cases where both the rule and retrieval tiers abstain, does RRPO improve over monolithic DPO and an untuned model given identical retrieval context? Please account for seed variability, including the collapsed runs reported in Tab8. Are there additional solver re-evaluations of optimized three-dimensional designs that confirm the reported objective improvements? These would materially affect my assessment of the inverse-design contributions.
6 comments
Response to Reviewer tEDX
W1: The retention comparison changes the routing architecture (Fig. 6)
We thank the reviewer for this sharp observation. To separate routing from the tuning corpus, we scored every adapter both bare and inside the deployed cascade, on the 24 CONSIST lessons (4 folds × 3 seeds).
| Lessons broken ↓ | Monolithic DPO | RRPO |
|---|---|---|
| Bare adapter, closed-book (Fig. 6a setting) | 16.8% | 12.6% |
| Deployed cascade | 0.4% | 0.4% |
- Retention comes from the routing. Inside the cascade both adapters break 2 of 549 eligible evaluation units (difference 0.0 points, 95% CI [−0.5, +0.5]). As bare adapters, monolithic DPO and RRPO break 16.8% and 12.6%, respectively.
With the rules and retrieval procedure fixed, decisions already resolved by those tiers are unaffected by adapter updates, allowing the residual tier to be updated while preserving consolidated decisions. We have moved Fig. 6 into the main text, where panel (a) shows this same-routing control; details are in Sec. 5.2 and Appendix G.3.
W1 & Q2: Unseen cases where both tiers abstain; untuned model; seed variability
We ran the experiment the reviewer describes, on the 35B base model of the arbiter.
- Unseen: four leave-one-group-out folds. All records of the held-out dataset group are removed from the tuning corpora and the precedent library.
- Both tiers abstain: the test set is every decision on which the statute tier has no legal answer and the precedent vote fails its mandate (15 two-option decisions, both option orders).
- Arms: monolithic DPO, RRPO, DPO on a random corpus of RRPO's size, SFT only, untuned. 3 seeds × 2 corpus sizes, 72 DPO runs, identical retrieval context.
| Open-book accuracy | RRPO | Monolithic | Random (same size) | SFT only | Untuned |
|---|---|---|---|---|---|
| One-third corpus | 62.2% | 57.8% | 55.6% | 53.3% | 53.3% |
| Full corpus | 66.7% | 62.2% | 60.0% | 53.3% | 53.3% |
- RRPO vs. monolithic DPO: RRPO achieves higher accuracy at both corpus sizes (62.2% vs. 57.8%; 66.7% vs. 62.2%). With the reduced corpus, RRPO already matches full-corpus monolithic DPO at 62.2%.
- RRPO vs. random corpus: RRPO exceeds size-matched random sampling at both sizes (62.2% vs. 55.6%; 66.7% vs. 60.0%). This comparison supports selecting pairs by rule coverage rather than sampling the same number at random.
- SFT-only and untuned models: both achieve 53.3%, while RRPO reaches 62.2% and 66.7%. These gains directly demonstrate the benefit of residual preference tuning on the held-out decisions that reach the learned tier.
- Training stability: 0 of 72 runs collapsed. The collapsed cells of Table 10 come from an exploratory sweep on a second, 8B base model.
We include these results in Sec. 5.2 and Appendix G.2, where RRPO is presented as the data-efficient way to train the residual tier.
Response to Reviewer tEDX
W2: The cambered AF-Panel case flips sign under the solver (Table 13)
To address this, we extended the original two-case solver validation to all 204 AF-Panel cases, each re-scored by the panel solver that labelled the dataset, with budgets relaxed to 2×, 5× and 10×. The optimization-and-solver audit ran on one GPU in about 1.25 hours.
| Budget | Surrogate ΔJ | Solver ΔJ | Prediction error (pp) |
|---|---|---|---|
| Deployed (1×) | −0.70% | −1.07% | 0.83 |
| 2× | −1.38% | −1.95% | 1.75 |
| 5× | −3.31% | −4.23% | 4.86 |
| 10× | −6.95% | −6.75% | 10.97 |
(Medians on a 164-case confirmation subset.)
- The solver confirms an improvement on 134 of 204 cases (65.7%). On the confirmation subset, the median solver gain (−1.07%) exceeds the surrogate's prediction (−0.70%).
- Prediction error grows with the budget, from 0.83 to 10.97 points. This supports the displacement budget used in deployment.
The expanded solver evaluations confirm objective improvements for most tested designs in both 2D and 3D, providing broader evidence for the optimizers' physical effectiveness. The AF-Panel results are in Appendix G.4.
W2 & Q3: No solver verification for the three-dimensional designs
To address this, we re-solved the deployed designs of two 3D products with RANS.
a. DrivAerNet++. Six validation cars were drawn by hash order and solved with OpenFOAM (simpleFoam, k-ω SST), the solver family behind the dataset labels. Including original, deployed and K5 designs, the 18 runs used 48 CPU cores each and about 1,175 CPU core-hours in total.
| Car | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
| Surrogate ΔC_D | −12.9% | −10.8% | −3.7% | −5.7% | −5.3% | −9.7% |
| OpenFOAM ΔC_D | −4.7% | −7.7% | −13.9% | −8.5% | −6.8% | −3.0% |
The solver confirms a drag reduction on all six cars, by 3.0% to 13.9% (7.4% on average).
b. BlendedNet++. We re-solved optimized designs with an independent RANS pipeline (OpenFOAM, rhoSimpleFoam + Spalart–Allmaras), calibrated against the dataset labels. Six validation starts were drawn by hash order; none of their geometries is in the training split. Each start is solved in five shapes: original, deployed design, a design with the displacement budget relaxed 5× (K5), and two random deformations of equal amplitude. All 30 solves converged, using 64 CPU cores per run and about 820 CPU core-hours in total.
| Start | Surrogate ΔC_D (deployed) | OpenFOAM ΔC_D (deployed) | OpenFOAM ΔC_D (K5) |
|---|---|---|---|
| 5892 | −28.6% | −15.0% | −13.8% |
| 3717 | −19.6% | −8.1% | −3.2% |
| 0740 | −9.0% | −4.1% | −20.1% |
| 0753 | −7.2% | −3.7% | −6.9% |
| 3057 | −10.9% | +1.3% | +9.3% |
| 2101 | −3.9% | +2.8% | +14.0% |
The results show that:
- The solver confirms the improvement on 4 of 6 starts, with drag reduced by 3.7% to 15.0% (noise floor 0.86%). On average the deployed designs change drag by −4.5%, against +3.5% for random deformations.
- The displacement budget limits adverse drag changes. On the other two starts the deployed designs stay within +1.3% and +2.8% of the original drag, whereas the relaxed budget moves them to +9.3% and +14.0%. This is the role the budget is designed for.
Solver alignment. BlendedNet++ was labelled with FUN3D. We calibrated an independent OpenFOAM pipeline against those labels using 193 solves at 64 CPU cores each (about 4,100 CPU core-hours); with the 30 evaluation solves, this totals 223 RANS solves and about 4,900 CPU core-hours. On unmodified geometries, the solvers agree on the direction of geometry-induced drag differences in 37 of 41 calibration pairs. The reported improvements compare each optimized design with its original geometry under identical solver settings and flow conditions.
c. AeroCar. The source work of the dataset (Liu et al., ICML 2026) re-simulated 500 surrogate-optimized FFD designs with GPU-accelerated LBM, and all 500 reduced drag (13.7% on average; Appendix B.1). This provides independent evidence for surrogate-guided FFD on AeroCar, complementing our direct solver evaluations of Caelum's DrivAerNet++ and BlendedNet++ designs.
We include these results in Sec. 5.2 (Table 3) and Appendix G.5, G.6.
Response to Reviewer tEDX
Thank you very much for the precise and constructive review. Below we respond to each point.
W1: Limited separation from Zhao et al. (2026)
We thank the reviewer for pointing to this work. Sec. 2.2 further clarifies the differences in task, supervision and overall framework:
RRPO is the residual-learning component of this framework. Our contributions are the end-to-end compiler, the build-decision corpus produced by counterfactual sweeps, and the consolidation mechanism that makes measured build expertise reusable.
W1: Both boards come from the same campaign
To measure behaviour on arriving datasets, we replayed all 69 scorable decisions of the campaign in arrival order, re-running legislation on the strict prefix, so that each dataset is answered only from what existed before it.
Reusable build expertise. As build experience accumulates, rule-and-precedent coverage rises from 2% to 79%, with 22 of 23 covered decisions answered correctly overall. This shows that counterfactual build evidence becomes reusable decision knowledge, allowing later datasets to benefit directly from earlier builds. We have clarified in Sec. 5 that the boards evaluate the reuse of consolidated experience, and added the replay as the evaluation on arriving datasets (Fig. 4(b); Appendix G.1).
Q1: Chronology for SuperWing, HiLift and AeroCar
Yes. The order was: profile the dataset → freeze the policy → write its predicted decisions to disk with a hash → open the build grid. Appendix G.1 gives the freeze record, and Fig. 4(c) shows the full build grids and final deployed configurations.
The frozen policy issued all 52 decisions for the three datasets and the HiLift-30 subset (13 decisions for each of four data settings; 49 by statute, 3 by the weight tier) before any build was trained.