acceptodds
Under review as a conference paper at ICLR 2027

Caelum: Self-Evolving Compilation of Forward–Inverse CAE Optimizers

Abstract

Inverse design is the promise of learned physics surrogates: differentiate through the surrogate, and simulation data becomes better geometry. Realizing it on a new dataset still demands a hand-built pipeline and dozens of expert decisions. The common remedy is to pretrain one large model across regimes. Yet CAE data is proprietary and rarely poolable, so specializing to each dataset is the more practical path. We present Caelum, a self-evolving compiler for this setting: from a raw CAE dataset it emits one deployable forward–inverse optimizer that predicts physical fields and reshapes geometry through them, specialized to its dataset and never leaving its owner. Our key insight is that build expertise is itself learnable. A simulation label costs a solver run, but an alternative build decision costs only retraining on data already in hand, so counterfactual build sweeps turn fifteen datasets into hundreds of process-rewarded preference pairs. Confirmed lessons harden into rules and precedents, and rule-residual preference optimization (RRPO) then tunes the language model only on decisions no rule can answer. Caelum ships every optimizer end-to-end across fifteen datasets from four physics domains. On frozen decision boards it answers every regime probe that frontier models miss and cuts the decision loss from 22.4 R2 points to 0.1. Design-loop construction becomes a self-evolving compilation process.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Ai4CAE, AI4Physics, Agent, Self-evolving Agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tEDXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents Caelum which constructs forward surrogates and differentiable inverse-design pipelines from CAE datasets. It collects preferences through controlled build sweeps, consolidates experience into explicit rules and retrieved precedents, and applies DPO to examples not covered by rules. The system is demonstrated across fifteen datasets in four physics domains, with evaluations of build decisions, forward predictions, and mostly surrogate-evaluated design optimization.

Critical strengths

This paper tries to address a practical CAE engineering bottleneck and implements the complete workflow across fifteen datasets. The controlled build sweeps reveal substantial changes in model rankings across data regimes (Sec. 4.4; Fig. 4). This empirical evidence has practical value for constructing surrogate pipelines and gives the work more substance than isolated agent demonstrations.

Critical weaknesses

There are two main cons, mainly regarding the evaluation and the claim:

First, the central learning contribution is not established by the evaluation. RRPO applies standard DPO to examples not covered by rules (Sec. 4.3), with limited methodological separation from existing rule-guided training, such as that of Zhao et al. (2026). More decisively, Appendix C states that all seven Board B decisions are resolved before the LLM tier, while precedent voting without an LLM already answers all fourteen statute-silent Board A items. Swapping between monolithic and residual adapters changes no deployed decisions (Table 9). Both boards also come from the same campaign used to construct the rules and precedents (Sec. 5). These results demonstrate recovery of accumulated build experience but do not establish transfer to unseen datasets or an additional benefit from RRPO.

The retention comparison likewise changes the routing architecture, comparing a bare adapter against the protected cascade (Fig. 6). A controlled evaluation on unseen tasks that actually require the learned tier is needed to substantiate the proposed learning contribution.

Second, the physical inverse-design capability is still insufficiently validated for me. Most improvements are evaluated by the same surrogate used to optimize the geometry (Sec. 5.2; Appendices D–E). Only two AF-Panel examples are checked using their source solver. For the cambered airfoil, a predicted 0.69% objective reduction becomes a 0.21% increase under solver evaluation, even though the design respects the displacement budget (Table 11). The discrepancy is small but demonstrates that bounded deformation alone does not validate the direction of improvement.

No corresponding solver verification is provided for the main three-dimensional design results. Producing improved physical designs is a central output of the proposed compiler, so this gap materially limits the claimed contribution.

Questions for authors

Were SuperWing, HiLift, and AeroCar evaluated with the rules, precedent library, and adapter frozen before any of their build outcomes entered the system? Please clarify the chronology and dataset-level separation.

On unseen cases where both the rule and retrieval tiers abstain, does RRPO improve over monolithic DPO and an untuned model given identical retrieval context? Please account for seed variability, including the collapsed runs reported in Tab8. Are there additional solver re-evaluations of optimized three-dimensional designs that confirm the reported objective improvements? These would materially affect my assessment of the inverse-design contributions.

6 comments
AuthorsOpenReview ↗

Response to Reviewer tEDX

Thank you very much for the precise and constructive review. Below we respond to each point.

W1: Limited separation from Zhao et al. (2026)

We thank the reviewer for pointing to this work. Sec. 2.2 further clarifies the differences in task, supervision and overall framework:

  • Problem: they learn a world model that predicts state transitions in text environments. Caelum decides how to build a CAE surrogate and its design loop.
  • Supervision: they learn from environment-interaction trajectories. Caelum uses counterfactual sweeps that change one build decision at a time, converting measured downstream performance differences into decision-level preference supervision.
  • Overall framework: Caelum consolidates a build-decision corpus into rules and precedents and combines them with residual preference learning in a decision cascade. This supports the complete compilation of private simulation data into deployable forward–inverse optimizers.

RRPO is the residual-learning component of this framework. Our contributions are the end-to-end compiler, the build-decision corpus produced by counterfactual sweeps, and the consolidation mechanism that makes measured build expertise reusable.

W1: Both boards come from the same campaign

To measure behaviour on arriving datasets, we replayed all 69 scorable decisions of the campaign in arrival order, re-running legislation on the strict prefix, so that each dataset is answered only from what existed before it.

Datasets (arrival order)Decisions answered by rules or precedentsCorrect among covered decisions
First 5 to arrive1 of 411 of 1
Later arrivals22 of 2821 of 22

Reusable build expertise. As build experience accumulates, rule-and-precedent coverage rises from 2% to 79%, with 22 of 23 covered decisions answered correctly overall. This shows that counterfactual build evidence becomes reusable decision knowledge, allowing later datasets to benefit directly from earlier builds. We have clarified in Sec. 5 that the boards evaluate the reuse of consolidated experience, and added the replay as the evaluation on arriving datasets (Fig. 4(b); Appendix G.1).

Q1: Chronology for SuperWing, HiLift and AeroCar

Yes. The order was: profile the dataset → freeze the policy → write its predicted decisions to disk with a hash → open the build grid. Appendix G.1 gives the freeze record, and Fig. 4(c) shows the full build grids and final deployed configurations.

AssetBefore any build of the three datasetsAfter
StatutesFrozen and hashedNone added
Precedent libraryFrozen at 213 records, none from these datasets16 records absorbed (213 → 229)
AdapterFrozen and hashedNot retrained

The frozen policy issued all 52 decisions for the three datasets and the HiLift-30 subset (13 decisions for each of four data settings; 49 by statute, 3 by the weight tier) before any build was trained.

AuthorsOpenReview ↗

Response to Reviewer tEDX

W1: The retention comparison changes the routing architecture (Fig. 6)

We thank the reviewer for this sharp observation. To separate routing from the tuning corpus, we scored every adapter both bare and inside the deployed cascade, on the 24 CONSIST lessons (4 folds × 3 seeds).

Lessons broken ↓Monolithic DPORRPO
Bare adapter, closed-book (Fig. 6a setting)16.8%12.6%
Deployed cascade0.4%0.4%
  • Retention comes from the routing. Inside the cascade both adapters break 2 of 549 eligible evaluation units (difference 0.0 points, 95% CI [−0.5, +0.5]). As bare adapters, monolithic DPO and RRPO break 16.8% and 12.6%, respectively.

With the rules and retrieval procedure fixed, decisions already resolved by those tiers are unaffected by adapter updates, allowing the residual tier to be updated while preserving consolidated decisions. We have moved Fig. 6 into the main text, where panel (a) shows this same-routing control; details are in Sec. 5.2 and Appendix G.3.

W1 & Q2: Unseen cases where both tiers abstain; untuned model; seed variability

We ran the experiment the reviewer describes, on the 35B base model of the arbiter.

  • Unseen: four leave-one-group-out folds. All records of the held-out dataset group are removed from the tuning corpora and the precedent library.
  • Both tiers abstain: the test set is every decision on which the statute tier has no legal answer and the precedent vote fails its mandate (15 two-option decisions, both option orders).
  • Arms: monolithic DPO, RRPO, DPO on a random corpus of RRPO's size, SFT only, untuned. 3 seeds × 2 corpus sizes, 72 DPO runs, identical retrieval context.
Open-book accuracyRRPOMonolithicRandom (same size)SFT onlyUntuned
One-third corpus62.2%57.8%55.6%53.3%53.3%
Full corpus66.7%62.2%60.0%53.3%53.3%
  • RRPO vs. monolithic DPO: RRPO achieves higher accuracy at both corpus sizes (62.2% vs. 57.8%; 66.7% vs. 62.2%). With the reduced corpus, RRPO already matches full-corpus monolithic DPO at 62.2%.
  • RRPO vs. random corpus: RRPO exceeds size-matched random sampling at both sizes (62.2% vs. 55.6%; 66.7% vs. 60.0%). This comparison supports selecting pairs by rule coverage rather than sampling the same number at random.
  • SFT-only and untuned models: both achieve 53.3%, while RRPO reaches 62.2% and 66.7%. These gains directly demonstrate the benefit of residual preference tuning on the held-out decisions that reach the learned tier.
  • Training stability: 0 of 72 runs collapsed. The collapsed cells of Table 10 come from an exploratory sweep on a second, 8B base model.

We include these results in Sec. 5.2 and Appendix G.2, where RRPO is presented as the data-efficient way to train the residual tier.

AuthorsOpenReview ↗

Response to Reviewer tEDX

W2: The cambered AF-Panel case flips sign under the solver (Table 13)

To address this, we extended the original two-case solver validation to all 204 AF-Panel cases, each re-scored by the panel solver that labelled the dataset, with budgets relaxed to 2×, 5× and 10×. The optimization-and-solver audit ran on one GPU in about 1.25 hours.

BudgetSurrogate ΔJSolver ΔJPrediction error (pp)
Deployed (1×)−0.70%−1.07%0.83
2×−1.38%−1.95%1.75
5×−3.31%−4.23%4.86
10×−6.95%−6.75%10.97

(Medians on a 164-case confirmation subset.)

  • The solver confirms an improvement on 134 of 204 cases (65.7%). On the confirmation subset, the median solver gain (−1.07%) exceeds the surrogate's prediction (−0.70%).
  • Prediction error grows with the budget, from 0.83 to 10.97 points. This supports the displacement budget used in deployment.

The expanded solver evaluations confirm objective improvements for most tested designs in both 2D and 3D, providing broader evidence for the optimizers' physical effectiveness. The AF-Panel results are in Appendix G.4.

W2 & Q3: No solver verification for the three-dimensional designs

To address this, we re-solved the deployed designs of two 3D products with RANS.

a. DrivAerNet++. Six validation cars were drawn by hash order and solved with OpenFOAM (simpleFoam, k-ω SST), the solver family behind the dataset labels. Including original, deployed and K5 designs, the 18 runs used 48 CPU cores each and about 1,175 CPU core-hours in total.

Car123456
Surrogate ΔC_D−12.9%−10.8%−3.7%−5.7%−5.3%−9.7%
OpenFOAM ΔC_D−4.7%−7.7%−13.9%−8.5%−6.8%−3.0%

The solver confirms a drag reduction on all six cars, by 3.0% to 13.9% (7.4% on average).

b. BlendedNet++. We re-solved optimized designs with an independent RANS pipeline (OpenFOAM, rhoSimpleFoam + Spalart–Allmaras), calibrated against the dataset labels. Six validation starts were drawn by hash order; none of their geometries is in the training split. Each start is solved in five shapes: original, deployed design, a design with the displacement budget relaxed 5× (K5), and two random deformations of equal amplitude. All 30 solves converged, using 64 CPU cores per run and about 820 CPU core-hours in total.

StartSurrogate ΔC_D (deployed)OpenFOAM ΔC_D (deployed)OpenFOAM ΔC_D (K5)
5892−28.6%−15.0%−13.8%
3717−19.6%−8.1%−3.2%
0740−9.0%−4.1%−20.1%
0753−7.2%−3.7%−6.9%
3057−10.9%+1.3%+9.3%
2101−3.9%+2.8%+14.0%

The results show that:

  • The solver confirms the improvement on 4 of 6 starts, with drag reduced by 3.7% to 15.0% (noise floor 0.86%). On average the deployed designs change drag by −4.5%, against +3.5% for random deformations.
  • The displacement budget limits adverse drag changes. On the other two starts the deployed designs stay within +1.3% and +2.8% of the original drag, whereas the relaxed budget moves them to +9.3% and +14.0%. This is the role the budget is designed for.

Solver alignment. BlendedNet++ was labelled with FUN3D. We calibrated an independent OpenFOAM pipeline against those labels using 193 solves at 64 CPU cores each (about 4,100 CPU core-hours); with the 30 evaluation solves, this totals 223 RANS solves and about 4,900 CPU core-hours. On unmodified geometries, the solvers agree on the direction of geometry-induced drag differences in 37 of 41 calibration pairs. The reported improvements compare each optimized design with its original geometry under identical solver settings and flow conditions.

c. AeroCar. The source work of the dataset (Liu et al., ICML 2026) re-simulated 500 surrogate-optimized FFD designs with GPU-accelerated LBM, and all 500 reduced drag (13.7% on average; Appendix B.1). This provides independent evidence for surrogate-guided FFD on AeroCar, complementing our direct solver evaluations of Caelum's DrivAerNet++ and BlendedNet++ designs.

We include these results in Sec. 5.2 (Table 3) and Appendix G.5, G.6.

Reviewer_RZNSOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents Caelum, a framework that automatically builds forward surrogate models and inverse design pipelines from CAE datasets. The main idea is to learn from previous build experiments rather than relying entirely on expert-defined configurations or LLM-generated decisions. The authors evaluate the system on 15 datasets from four physics domains. The results show substantial improvements in build decision quality over zero-shot LLM baselines. The paper also demonstrates forward prediction and surrogate-based inverse optimization, and reports that RRPO can preserve the deployed decisions while using a much smaller preference dataset.

Critical strengths
  1. End-to-end automated CAE optimization framework: Caelum automates interconnected decisions ather than treating surrogate training as an isolated problem. Experiments across 15 datasets and four physics domains demonstrate that it can construct executable optimization pipelines, making it convincing as an engineering framework.
  2. The framework uses counterfactual build sweeps to generate preference data from measured performance differences. Accumulated experience is organized into explicit rules, retrieved precedents, and an LLM policy, while Rule‑Residual Preference Optimization (RRPO) focuses training on decisions not covered by existing rules.
Critical weaknesses
  1. Limited evaluation of generalization to unseen datasets: Both decision evaluation boards are constructed from the same experimental campaign used to establish rules and precedents. Consequently, the strong decision accuracy mainly reflects the framework’s ability to reuse accumulated experience rather than its ability to handle previously unseen tasks.
  2. Insufficient evaluation of RRPO in its intended residual decision region: RRPO is designed to improve decisions when neither explicit rules nor precedent retrieval can provide an answer. However, the experiments do not directly evaluate this region—Board A cases are statute‑silent but resolved by precedents, and Board B avoids LLM invocation. Table 8 reports improvements, but these do not establish RRPO’s effectiveness when both tiers abstain, leaving its independent contribution unclear.
  3. Limited evidence of physical improvement in inverse design: Most reported improvements are evaluated using the same surrogate models that guide optimization, risking surrogate bias. Solver‑based validation is provided for only two AF‑Panel cases, one of which shows physical deterioration despite surrogate‑predicted improvement. This weakens evidence that the generated optimizers consistently improve physical design objectives.
Questions for authors
  1. Generalization Capability The evaluation boards are constructed from the same experimental campaign used to build rules and precedents, which may overestimate generalization. Could you provide results on completely held‑out datasets where no prior rules/precedents exist?

  2. Independent Contribution of RRPO RRPO is specifically designed for residual decisions where explicit rules and precedent retrieval abstain. However, the current experiments (Board A/B) do not isolate this region—Board A cases are resolved by precedents, and Board B avoids LLM invocation.

  3. Physical Validation Most inverse design improvements are evaluated using the same surrogate that guides optimization, which risks surrogate bias. Given that Appendix D shows one of two solver‑validated cases deteriorated physically, could you expand solver‑based validation to more tasks across all four physics domains?

4 comments
AuthorsOpenReview ↗

Response to Reviewer RZNS

Thank you very much for the thorough review and for recognizing Caelum as an end-to-end engineering framework. Below we respond to each point.

W1 & Q1: Generalization to unseen datasets

We address this at two levels.

a. The three datasets added last were handled prospectively. For SuperWing, HiLift and AeroCar, the statutes, the precedent library and the adapter were frozen and hashed, and the policy's 52 decisions for the three datasets and the HiLift-30 subset (13 per data setting) were written to disk before any build was trained. Fig. 4(c) shows the full build grids for the three full datasets.

b. Arrival-order replay of the whole campaign. We replayed all 69 scorable decisions in the order the datasets arrived, re-running legislation on the strict prefix, so that each dataset is answered only from what existed before it.

Datasets (arrival order)Decisions answered by rules or precedentsCorrect among covered decisions
First 5 to arrive1 of 411 of 1
Later arrivals22 of 2821 of 22

Reusable build expertise. As build experience accumulates, rule-and-precedent coverage rises from 2% to 79%, with 22 of 23 covered decisions answered correctly overall. Because each dataset is answered using only earlier evidence, this demonstrates how measured build expertise transfers to later datasets through rules and precedents (Fig. 4(b); Appendix G.1).

W2 & Q2: RRPO is not evaluated in its residual region

To address this, we evaluated exactly this region under dataset-level holdout (four leave-one-group-out folds; all records of the held-out group removed from the tuning corpora and the precedent library). The test set is every decision on which the statute tier has no legal answer and the precedent vote fails its mandate: 15 two-option decisions. We ran 72 DPO runs on the 35B arbiter model (3 arms × 2 corpus sizes × 3 seeds × 4 folds).

Open-book accuracyRRPOMonolithicRandom (same size)SFT onlyUntuned
One-third corpus62.2%57.8%55.6%53.3%53.3%
Full corpus66.7%62.2%60.0%53.3%53.3%
  • RRPO leads all four baselines at both corpus sizes, reaching 62.2% and 66.7%, compared with 57.8% and 62.2% for monolithic DPO and 53.3% for both SFT-only and untuned models.
  • Rule-based selection exceeds random selection at equal size: RRPO achieves 62.2% vs. 55.6% with the smaller corpus and 66.7% vs. 60.0% with the full corpus.
  • Data efficiency: RRPO with the reduced corpus matches full-corpus monolithic DPO at 62.2%. The comparison therefore supports both the quality of residual selection and its efficient use of preference data.
  • Training stability: 0 of 72 runs collapsed.

These results directly support RRPO's contribution to decision accuracy and data efficiency in its intended residual region. Retention comes from the cascade: in a same-routing control, monolithic and RRPO adapters both break 0.4% of mastered lessons inside the cascade, against 12.6% to 16.8% as bare adapters (Appendix G.2, G.3).

AuthorsOpenReview ↗

Response to Reviewer RZNS

W3 & Q3: Limited evidence of physical improvement

To address this, we expanded solver validation from 2 cases to all 204 AF-Panel cases, and added 48 RANS solves on two 3D products (about 2,000 CPU core-hours).

a. AF-Panel, all 204 cases (re-scored by the panel solver that labelled the dataset; the optimization-and-solver audit used one GPU for about 1.25 hours):

  • The solver confirms an improvement on 134 of 204 cases (65.7%).
  • On the 164-case confirmation subset, the median solver gain is −1.07%, larger than the surrogate's prediction of −0.70%.
  • Prediction error grows from 0.83 points at the deployed budget to 10.97 points at a 10× budget, which supports the displacement budget used in deployment.

b. DrivAerNet++, three-dimensional RANS (six validation cars drawn by hash order; OpenFOAM simpleFoam, k-ω SST, the solver family behind the dataset labels). Including original, deployed and K5 designs, the 18 runs used 48 CPU cores each and about 1,175 CPU core-hours in total:

Car123456
Surrogate ΔC_D−12.9%−10.8%−3.7%−5.7%−5.3%−9.7%
OpenFOAM ΔC_D−4.7%−7.7%−13.9%−8.5%−6.8%−3.0%
  • The solver confirms a drag reduction on all six cars, by 3.0% to 13.9% (7.4% on average).

c. BlendedNet++, three-dimensional RANS (OpenFOAM; six validation starts drawn by hash order; original, deployed, 5×-relaxed budget K5 and two random shapes per start; 30 converged solves, 64 CPU cores per run, about 820 CPU core-hours in total):

StartSurrogate ΔC_D (deployed)OpenFOAM ΔC_D (deployed)OpenFOAM ΔC_D (K5)
5892−28.6%−15.0%−13.8%
3717−19.6%−8.1%−3.2%
0740−9.0%−4.1%−20.1%
0753−7.2%−3.7%−6.9%
3057−10.9%+1.3%+9.3%
2101−3.9%+2.8%+14.0%
  • The solver confirms drag reduction on 4 of 6 starts (3.7% to 15.0%).
  • On the other two starts the deployed designs stay within +1.3% and +2.8% of the original drag, against +9.3% and +14.0% under the relaxed budget, which is the role the budget is designed for.

d. Validation scope and solver alignment. The new solver evidence covers 2D and 3D aerodynamics, including two types of 3D product; extending these checks to the remaining domains requires their corresponding solver setups. For FUN3D-labelled BlendedNet++, we calibrated an independent OpenFOAM pipeline using 193 solves at 64 CPU cores each (about 4,100 CPU core-hours); with 30 evaluation solves, this totals 223 solves and about 4,900 CPU core-hours. The solvers agree on the direction of drag differences in 37 of 41 geometry pairs. All optimization results compare original and optimized shapes under identical solver settings and flow conditions. The AeroCar source work (Liu et al., ICML 2026) additionally reports drag reduction in all 500 surrogate-guided FFD designs re-evaluated with GPU-accelerated LBM (13.7% on average), supporting that optimization route.

We include these results in Sec. 5.2 (Table 3) and Appendix G.4–G.6.

Reviewer_RZNSOpenReview ↗

Thanks for the detailed reply from the authors. I will keep my positive score.

Thanks for the detailed reply from the authors. I will keep my positive score.

AuthorsOpenReview ↗

Response to Reviewer RZNS

Thank you for reading our response and for keeping your positive score. Your comments on held-out generalization, RRPO, and solver-based validation have helped us improve the paper.

Reviewer_tKV8OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Disclaimer: I lack expertise in CAE and have alerted the AC. My review only assess the machine learning and agentic framework parts. I cannot reliably assess the physical modelling parts, or its validity and novelty in existing CAE field.

From my understanding, Caelum can fall into autoML category. The main task is to learn a deployable forward-reverse optimizer, which is a numerical prediction model, from a raw CAE dataset. In order to generate the model, it needs to make a sequence of discrete decisions, which are actually the hyperparameters of the model. It first uses rule-based decisions. When no rule can answer for a particular decision, it utilizes LLM to decide. It uses a method called counterfactual build sweep to generate training data as the raw CAE dataset is limited. This paper also proposes a new training method rule-residual preference optimization (RRPO), which is a slight modification of DPO, to finetune LLMs to provide better decision.

Critical strengths

This paper provides a concrete method called "counterfactual build sweep" to generate synthetic supervision data from limited raw CAE dataset. The counterfactual build sweep compares the downstream performance of configurations differ in one single decision, so that it can generate preference pairs with credit assignment. It then allows process reward construction, and LLM finetuning using this dataset.

Critical weaknesses

The contribution of the rule-residual preference optimization (RRPO) is not convincingly demonstrated in this paper. RRPO is presented as one of the three key contributions. However, the paper does not have experiments clearly showing the benefits of RRPO compared with models without finetuning or finetuned on other baseline methods.

Appendix C says "On the fourteen statute-silent board items both adapters score 10/10 and 4/4 open-book, as does the precedent vote with no language model at all. The adapters differ only closed-book, where the monolithic one answers more items, the expected return on its larger corpus. On Board B the deployed cascade answers all seven items at the statute and vote tiers". It seems that without LLM the pipeline can achieve the same results. Therefore, the benefits of LLM is not shown in this experiment, and it does not seem to be a correct experiment to demonstrate the usefulness of RRPO.

The only evidence is finetuning with RRPO achieves slightly better board accuracy compared to finetuning on the entire corpus in Table 8. However, 2 of the 3 seeds collapsed in that row of best results, showing the instability of RRPO. Also if we compare across all the configurations, finetuning on the entire corpus is usually better than RRPO.

In addition, full corpus DPO is the only baseline it compares to. It would be meaningful to compare to other fine tuning methods, such as standard DPO with a randomly sampled preference corpus of the same size as used in RRPO.

Questions for authors

Are there experiments showing benefits of RRPO more clearly? Is it possible to compare it with more baselines of other finetuning methods, such as DPO with a randomly sampled preference corpus of the same size as used in RRPO? Or would you consider RRPO more as a design choice in your pipeline instead of a main algorithmic contribution?

6 comments
AuthorsOpenReview ↗

Response to Reviewer tKV8

Thank you very much for the careful review of the learning and agentic parts. We first clarify the CAE build-decision problem and then explain the roles of the components.

Background: AI for CAE, and where Caelum sits

  • CAE. Computer-aided engineering evaluates a design by numerically solving its physics, for example the airflow around a car. One high-fidelity run takes hours to days.
  • Neural surrogates. Trained on solver outputs, a network predicts the physical field of a new geometry in one forward pass. This line runs from neural operators [1] to transformer solvers on industrial geometries [2, 3].
  • Inverse design. A surrogate is differentiable, so an objective such as drag can be back-propagated to the geometry, and the model can also reshape the design [4].
  • The bottleneck. Each new dataset requires choices of split, model family, capacity, objective and constraints. A shared pretrained model [5] does not directly resolve these build and deployment decisions for private simulation data; Caelum supplies measured evidence and compiles the optimizer on the owner's side.
  • LLM agents. They automate mainstream ML engineering [6], but build choices on private datasets still require the corresponding measured evidence. In our benchmark, frontier LLMs fall below chance on regime-dependent choices (Fig. 3b).

Our work. Caelum runs counterfactual build sweeps to manufacture that missing evidence, consolidates it into rules and precedents, and compiles each private dataset into a deployed forward–inverse optimizer. In AutoML terms, as the reviewer puts it, the hyperparameter decisions are learned from measured build outcomes.

[1] Li et al., Fourier Neural Operator, ICLR 2021. [2] Wu et al., Transolver, ICML 2024. [3] Luo et al., Transolver++, ICML 2025. [4] Remelli et al., MeshSDF, NeurIPS 2020. [5] Wu et al., GeoPT, ICML 2026. [6] Chan et al., MLE-bench, ICLR 2025.

W1: Benefits of RRPO over untuned and monolithic models

To address this, we ran the comparison the reviewer proposes, including DPO on a randomly sampled corpus of the same size as RRPO's.

  • Arms: RRPO, monolithic DPO, random-corpus DPO, SFT only, no fine-tuning; on the 35B base model of the arbiter.
  • Test: held-out dataset groups (4 folds). The 15 test decisions are those on which both rules and precedents abstain, i.e., exactly the decisions that reach the LLM.
  • Runs: 3 seeds × 2 corpus sizes × 3 arms × 4 folds = 72 DPO runs, one fixed recipe.
AccuracyRRPOMonolithicRandom (same size)SFT onlyNo fine-tuning
One-third corpus62.2%57.8%55.6%53.3%53.3%
Full corpus66.7%62.2%60.0%53.3%53.3%
  • RRPO vs. monolithic DPO: RRPO achieves higher accuracy at both sizes (62.2% vs. 57.8%; 66.7% vs. 62.2%). The reduced-corpus RRPO model also matches full-corpus monolithic DPO at 62.2%, demonstrating data efficiency.
  • SFT-only and no fine-tuning: both baselines achieve 53.3%; RRPO improves accuracy to 62.2% and 66.7%. The learned tier thus adds measurable value on the decisions where both rules and precedents abstain.

These controlled results are reported in Sec. 5.2 and Appendix G.2.

AuthorsOpenReview ↗

Response to Reviewer tKV8

W2: Without the LLM the pipeline achieves the same results (Appendix C)

Thank you for pointing this out. The reviewer's reading is correct, and this is the intended behaviour of the system. The boards are written from experience that the cascade has already consolidated, so rules and precedents answer them on CPU in seconds. The LLM is the fallback for the rest. The following results make the roles explicit:

  • What the learned tier adds. On held-out decisions where both rules and precedents abstain, RRPO reaches 66.7%, versus 62.2% for monolithic DPO, 60.0% for size-matched random selection and 53.3% for SFT-only/untuned models (Appendix G.2).
  • How often the LLM is needed. Replaying the campaign in arrival order, rules and precedents answer 1 of 41 decisions on the first five datasets and 22 of 28 on the later arrivals (Appendix G.1).
  • What protects earlier lessons. A bare adapter breaks 12.6% to 16.8% of mastered lessons; inside the cascade, both adapters break 0.4% (Appendix G.3).

W3: Stability of the tuning results

  • Training stability: 0 of 72 runs collapsed. The collapsed seeds in Table 10 come from an exploratory sweep on a second, 8B base model.

W4: DPO on a size-matched random preference corpus

  • RRPO vs. random sampling: RRPO leads the size-matched baseline at both corpus sizes (62.2% vs. 55.6%; 66.7% vs. 60.0%). This directly addresses whether residual selection is more effective than simply sampling fewer pairs.

Q: Would you consider RRPO a design choice in the pipeline instead of a main algorithmic contribution?

RRPO is the component responsible for learning from residual preferences within Caelum. Our core contribution is a compiler framework that accumulates and reuses CAE build expertise to turn private simulation datasets into deployable forward–inverse optimizers. To achieve this, Caelum uses counterfactual build sweeps to turn measured differences between build choices into supervision for individual decisions, constructing a build-decision corpus. It then consolidates validated lessons into rules and precedents, so evidence produced by one build can inform subsequent builds. Within this framework, RRPO follows from how the knowledge is organized: rules handle the decisions they already cover, and only preference pairs outside that coverage are used for fine-tuning, concentrating the training budget on residual decisions. We have further clarified the relationship between this overall contribution and RRPO's role as a component in the revised contribution list and Sec. 4.3.

Presentation. For readers outside CAE, the revision adds Appendix F with worked examples (a trajectory, a preference pair, a statute, a retrieved precedent, board items), the rule-tier decision interface, and a glossary.

Thank you again for your thoughtful review. We understand that assessing work in a less familiar field can be challenging, and we would be glad if our response has provided helpful background and a clearer picture of CAE.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response Summary and Updated Manuscript

We sincerely thank all reviewers for their detailed and constructive feedback. In particular, we greatly appreciate the acknowledgements from:

  • Reviewer tKV8: for recognizing the counterfactual build sweep as "a concrete method" that generates supervision "with credit assignment" from limited CAE data.
  • Reviewer RZNS: for recognizing Caelum as an end-to-end framework that is "convincing as an engineering framework" across 15 datasets and four physics domains.
  • Reviewer tEDX: for recognizing that Caelum addresses "a practical CAE engineering bottleneck" with "more substance than isolated agent demonstrations".

Based on this feedback, we ran six new controlled studies and revised the paper. The concerns are addressed as follows:

ConcernReviewerNew evidenceLocation in Manuscript
Solver validation of inverse designs, including 3DRZNS, tEDX204 source-solver re-evaluations (2D) + 48 OpenFOAM RANS solves on two 3D products (about 2,000 CPU core-hours for the 3D evaluations)Sec. 5.2 (Table 3); Appendix G.4–G.6
Retention comparison changes routingtEDXSame-routing control, 4 folds × 3 seedsSec. 5.2 (Fig. 6); Appendix G.3
RRPO where both tiers abstain; baselines; seedstKV8, RZNS, tEDXHeld-out residual accuracy: RRPO 66.7%, monolithic 62.2%, size-matched random 60.0%, SFT-only/untuned 53.3%; 72 DPO runsSec. 5.2; Appendix G.2
Unseen datasets and chronologyRZNS, tEDXFreeze record + arrival-order replaySec. 5 (Table 1; Fig. 4); Appendix G.1
Relation to Zhao et al. (2026)tEDXRevised positioningSec. 2.2

Positioning. Caelum is a system that compiles a private CAE dataset into a deployed forward–inverse optimizer. Its contributions are the end-to-end compiler, the build-decision corpus manufactured by counterfactual sweeps, and the consolidation of that corpus into rules and precedents. RRPO is the component that adapts DPO to the small residual corpus. The contribution list now states this explicitly (Sec. 1).

Practical value. We would like to highlight what the system delivers in practice:

  • Dataset in, design tool out. An engineer submits a directory of solver results and, within hours, receives a deployed tool that predicts the physical field of a shape and redesigns it. The data never leaves its owner.
  • Build decisions grounded in measured evidence. On a single dataset, defensible build configurations differ by up to 94 R2R^2 points. All nine frontier LLMs score below a random pick on Board B, while the deployed cascade is within 0.11 R2R^2 points of the oracle (Table 2).

Scale of the work. To our knowledge, no build-decision corpus for CAE existed before this work, so every piece of evidence had to be produced by training:

  • Campaign: 15 datasets from four physics domains (about 30,000 solver samples over 24,000 geometries), 601 build-and-evaluate trajectories, 229 measured preference records, 13 arbitration decision queries, and 15 shipped forward–inverse products.
  • Rebuttal period: 72 DPO runs on a 35B model, 204 source-solver re-evaluations, and 241 RANS solves for the 3D studies (193 to calibrate a solver against dataset labels, 48 in the two evaluations; 48–64 CPU cores per run, about 6,100 CPU core-hours).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.