- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors present EMTopoBench-54, a nonlinear magnetostatics benchmark with various topology families and 500 samples per family. They also develop CoRaWS, an architecture for neural warm-starts of numerical solvers. They train on source families of their benchmark and transfer to unseen target topologies, while still showing an acceleration of the solving time.
Critical strengths
This paper introduces an interesting new benchmark to the machine learning community, which is itself valuable. The authors add that this benchmark is intended for evaluating the transferability of methods to unseen topologies at test time. The experimental results and ablations are thorough and demonstrate that CoRaWS is still capable of accelerating solving time even at topologies unseen during training.
Critical weaknesses
While the goal of introducing a new scientific benchmark is valuable to the machine learning community, a few additions to this paper would strengthen this. Currently, the benchmark is framed in the abstract as supporting “the study of compositional generalization through training on source families and evaluation on unseen target families.” A scientific benchmark with this purpose would be very valuable to the community. However, the focus of the paper (correct me if I am misunderstanding) is specifically for neural warm-start methods. As such, to present a more general benchmark, I would recommend including comparisons of existing trained surrogate models (no need to develop new ones here, as CoRaWS already does this for the warm-start setting), i.e., models without solvers-in-the-loop. This comparison would evaluate transferability to unseen topologies under the common surrogate modeling setting, and would strengthen the position of the benchmark substantially. The other option would be to constrain the settings where the benchmark should apply, specifically to neural warm-start methods. However, this runs the risk of limiting the reach and impact of the benchmark.
On a more minor note: Table 1 measures wall-clock time, but do you have any indication of how CoRaWS compares to other methods in terms of FLOPs and other timing metrics? I understand it may be computationally infeasible to recompute the precise FLOP count, but a rough estimate would be valuable because wall-clock comparisons could differ across machines.
While the paper is well-written, I believe there are some sections where machine learning audiences may struggle to understand. Concretely, I would recommend:
- Emphasizing why this benchmark is important and what it adds over previous benchmarks
- Modifying some of the background sections in the main text. Some sections (e.g., eq 1) are too dense and might benefit from simplifications or expanded explanations. For instance, if this background is just to determine that “We therefore predict in magnetic materials and recover through the inverse constitutive relation.”, then perhaps the background can be moved into the appendix and this result can be emphasized in the main text.
I am open to improving my score.
Questions for authors
Do multiple resolutions matter in this application? If so, using ConvNeXt style network (i.e., not operator) architectures may limit generalization to unseen resolutions. This may be important if the benchmark considered resolutions unseen during training, but I am not very familiar with the application domain.
6 comments
Response to Reviewer R2hw — Part 2/4
W2 | Network computation and numerical solver workload
Thank you for suggesting additional computational metrics. We quantify network forward computation and numerical solver work separately to explain the sources of timing differences. At a input resolution, CoRaWS's forward pass includes three-head prediction, magnitude classification, and H-to-B recovery. The PyTorch profiler counts are as follows:
| Model | Parameters (M) | Operations per forward pass (GFLOPs, partial count) |
|---|---|---|
| Direct-B | 30.678 | 26.050 |
| CoRaWS | 30.779 | 26.054 |
We additionally calculate normalization and bilinear interpolation costs from the actual tensor dimensions and estimate constitutive recovery costs using the 64-point material table. One multiply-add is counted as two floating-point operations.
| Operation | Direct-B (GFLOPs) | CoRaWS (GFLOPs) |
|---|---|---|
| LayerNorm / GroupNorm | Approximately 0.045 | Approximately 0.045 |
| Bilinear interpolation, including coordinate and weight calculations | Approximately 0.023–0.046 | Approximately 0.026–0.050 |
| Magnitude interpolation and vector recovery after constitutive lookup | — | Approximately 0.00059 |
Normalization is estimated at approximately 8 basic operations per element, and the interpolation range reflects whether coordinates and weights are reused across channels. For raster points, constitutive recovery additionally involves 65,536 square roots and at most 458,752 binary-search comparisons, with indexing, comparisons, and memory accesses not counted as multiply-adds. Interpolation here refers to upsampling within the network, while field transfer to FEM quadrature points is counted separately as initial-guess construction. After deducting the constitutive multiply-adds already counted, the supplemented arithmetic counts are approximately 26.12–26.14 GFLOPs for Direct-B and 26.13–26.15 GFLOPs for CoRaWS. Most computation still comes from the shared backbone. Special functions such as exponentials and logarithms cannot be interpreted in terms of ordinary multiply-add time, so the complete cost is still measured by online time.
Sparse FEM does not lend itself to a single FLOP count per update because the degrees of freedom, matrix structure, and factorization fill-in vary across samples, and line search requires repeated computation. We therefore separately count coarse- and fine-mesh updates and initial-guess constructions across all 1,440 pairs:
| Method | Fine-mesh Newton updates | Coarse-mesh Newton updates | Full network forward passes | B→Az projections |
|---|---|---|---|---|
| Zero initialization | 7,662 | 0 | 0 | 0 |
| Direct-B | 7,193 | 0 | 1,440 | 1,440 |
| CoRaWS, routing off | 6,773 | 0 | 1,440 | 1,440 |
| CoRaWS, routing on | 6,759 | 0 | 1,440 | 319 |
| C2F | 5,597 | 15,706 | 0 | 0 |
| C2F, same routing | 5,805 | 12,882 | 1,440 | 0 |
CoRaWS reduces fine-mesh updates by 11.79% relative to zero initialization. Routing still performs 1,440 full forward passes but reduces projections from 1,440 to 319, avoiding construction overhead on easy samples. C2F instead trades coarse-mesh solution and transfer for fewer fine-mesh updates. C2F without routing includes 2,122 coarse-mesh refinements, compared with 957 under the same routing. These metrics complement wall-clock time by showing how computational work is reduced or shifted, and will be reported in Appendix E.5.
Response to Reviewer R2hw — Part 3/4
W3 | Presenting the benchmark's value and the method's intuition
Following your suggestion, we will present the benchmark's value and the method's intuition before the details. The benchmark section will first explain the evaluation capabilities it adds, and the method section will explain why we predict H before developing the structural definitions and mathematical analysis. Readers can thus first understand the problem and design objectives, with detailed derivations available in the appendix.
The benchmark's main value is to explicitly distinguish parameter changes from changes in assembly structure. Varying dimensions or excitation while keeping component combinations and assembly relations fixed presents the model with parameter changes within the same structure. Changing component combinations and assembly relations, however, can alter magnetic circuit connections and branches, requiring the model to transfer to new structures. EMTopoGen explicitly defines primitives, assembly relations, and continuous parameters, on which EMTopoBench-54 builds 54 topology families and 27,000 samples and evaluates structural transfer by holding out entire families.
We will emphasize this evaluation dimension in the introduction and benchmark description and use the new FNO and GAOT results to show that the same benchmark can evaluate both field prediction on unknown structures and whether those predictions assist numerical solution. The structural grouping in the appendix further distinguishes unseen families within seen engineering groups, unseen engineering groups composed of seen primitives, and target families containing unseen primitives, allowing readers to see the difficulty and outcomes of each type of transfer.
For the method derivation, Section 3.2 will first explain the intuition: at different magnetization states, the same B deviation can correspond to different H deviations, so the material constitutive relation provides a physically motivated way to measure error. We will then explain why we predict H in magnetic materials and recover B through the known constitutive relation. The scalar expansion and detailed derivations are moved to Appendix C.1, while the main text retains the vector form and its connection to the FEM tangent response. Section 3.3 will likewise explain the roles of encoding, magnitude classification, and the prediction heads before introducing the equations.
We believe these changes preserve the theoretical basis of the method while allowing readers to first understand its design objectives. The local analysis explains how the representation changes error weighting, while experiments test whether it reduces solver work.
Response to Reviewer R2hw — Part 1/4
Dear Reviewer R2hw,
Thank you for suggesting that we demonstrate the benchmark's use in conventional surrogate modeling, and for the encouragement of your willingness to consider raising your score; we will carefully follow your suggestions to further improve the paper. We have added cross-topology-family direct prediction with FNO and GAOT and stratification by field magnitude using a common criterion, quantified network computation and numerical solution work, and tested unseen input resolutions. We will use these additions to strengthen the benchmark introduction and present the physical intuition and evaluation objectives before the mathematical details. We have completed the relevant revisions in response to your suggestions, and we hope that these changes and additional evidence fully address your concerns and encourage you to reconsider your assessment of our work. If any issues still affect your judgment, we would be happy to clarify them further and make additional improvements.
W1 | The benchmark's use with conventional surrogate models
Thank you for suggesting that we validate the benchmark's uses separately from CoRaWS itself. We have added conventional field-prediction experiments with FNO and the geometry-aware operator GAOT, directly evaluating their transfer to unseen topology families without a numerical solver during prediction. The geometries, materials, excitations, field labels, and topology-family splits provided by EMTopoBench-54 support this evaluation. We have added a dedicated section to introduce this, and it does have potential for application to other domains.
All models use the same three splits, with 12,000 source-family training samples, 120 epochs, and 480 target-family pairs per split. FNO and GAOT select between linear and radial asinh output representations on 600 held-out source-family samples, training both candidates for 12 epochs and continuing the selected candidate to 120 epochs, with training of the unselected candidate counted separately as selection cost.
To better understand which samples account for these errors, we stratify by ground-truth field RMS within the same valid region V:
Brms∗=∣V∣1∑x∈V∥B∗(x)∥22.
With thresholds of 1 and 100 mT, all models use exactly the same samples.
The stratified results explain why the overall median is high: 1,164 pairs have a ground-truth field RMS below 100 mT, accounting for 80.83% of all pairs. These samples are numerous and have high relative errors, so the overall median primarily reflects prediction performance at lower field magnitudes. By contrast, CoRaWS has a median error of 16.63% in the group with a ground-truth field RMS of at least 100 mT. In fact, relative error uses the ground-truth field norm as its denominator, so small absolute deviations in weak fields can correspond to large relative errors. For example, in the group below 1 mT, CoRaWS's median absolute field-error RMS is approximately 0.392 mT. Although small in absolute terms, this deviation is not negligible relative to the actual field.
Following your suggestion, we have added direct evidence of the benchmark's use in conventional surrogate modeling, revealing prediction challenges at different field magnitudes. Direct-B has lower errors in the two weak-field groups, while CoRaWS's H recovery achieves a lower error in the group at or above 100 mT. This helps explain why routing in the original manuscript can more effectively reduce subsequent solver work in warm starts. It also allows the same benchmark to evaluate both field prediction and the assistance predictions provide to numerical solution. We will therefore retain EMTopoBench-54's focus on compositional generalization, add conventional surrogate-model comparisons in Section 4.2, and report the common field-magnitude stratification and training and evaluation settings in Appendix E.4.