acceptodds
Under review as a conference paper at ICLR 2027

CLOAKFEM: Agentic FEM Code Generation for Red-Teaming Computational Resource Exhaustion

Abstract

Finite element method (FEM) solvers are foundational to scientific and engineering simulation, yet their susceptibility to efficiency attacks remains largely unexplored. An adversary can craft syntactically valid and runnable FEM programs that complete successfully while forcing excessive computational work, enabling computational denial-of-service (DoS) attacks. Finding such inputs is challenging because FEM programs occupy a vast, structured space constrained by numerical, geometric, and constitutive requirements. We introduce CloakFEM, a domain-guided LLM-agent framework for automatically generating runnable adversarial FEM programs that amplify solver computation. CloakFEM combines LLM-guided search with an FEM-specific intermediate representation and a deterministic validity gate that enforces runnable constraints. Across three FEM stacks, 17 problem classes, and 1,068 seeds, CloakFEM discovers adversarial inputs that substantially amplify solver work while remaining valid and solvable. In the most extreme case, solver iterations increase from 43 to 247,367, a 5,752.72× amplification. CloakFEM outperforms five generic black-box baselines and retains an advantage over evolutionary search under a controlled bounded action space. Cross-solver analysis further reveals both problem-intrinsic and solver-specific sources of vulnerability. Our results uncover an underexplored efficiency attack surface in FEM solvers and show that runnable FEM programs can serve as adversarial inputs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Code Generation, LLM Agent, Adversarial Attack, FEM Testing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DE4wOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies computational resource exhaustion attacks against finite element method (FEM) solvers and proposes CloakFEM, a framework that uses LLM agents to automatically generate FEM programs with high computational costs. The method uses a unified intermediate representation (IR) to describe modifiable FEM parameters and combines three agents—Strategy, Generation, and Selection—to iteratively search for inputs that significantly increase computational costs based on solver feedback. It also uses validity checks to ensure that the generated programs can run and complete the solution process.

The authors conduct experiments across three FEM environments, 17 problem categories, and 1,068 initial samples, reporting a maximum amplification of 5,752.72× in solver iteration counts. The method is compared with five conventional black-box search methods. In addition, the paper includes comparisons under a restricted search space, component ablations, cross-solver transfer experiments, and defense experiments.

Critical strengths

S1. The research problem has some exploratory value.

This paper focuses on the phenomenon that valid inputs can significantly increase the computational costs of FEM solvers and studies this problem from the perspective of computational resource exhaustion attacks. It provides a direction worth exploring for evaluating the security of traditional numerical computing software.

S2. An automated search framework across different FEM solvers is implemented.

The authors design a unified FEM intermediate representation and a validity-checking mechanism, allowing the framework to handle inputs from different FEM environments and automatically search for computationally expensive programs based on feedback from actual solver executions. This has some engineering value.

Critical weaknesses

W1. Limited methodological novelty, and the proposed multi-agent workflow lacks substantial technical contributions.

The core workflow of CloakFEM uses three agents to propose modification directions, generate candidates, and select candidates, followed by iterative optimization based on solver feedback. This basically follows the common paradigm of existing LLM-agent-based search and optimization methods. The FEM-specific IR and validity checks are more like domain-specific engineering adaptations rather than new search algorithms or optimization mechanisms. Therefore, I think the methodological novelty of this paper is relatively limited.

W2. The practical necessity and application value of the method are unclear.

The paper constructs inputs with high computational costs through parameter search, but the main mechanisms it uses, such as near-incompressibility, material coefficient contrast, and mesh distortion, are all known causes of numerical difficulties. Section 4.3 also attributes some results to the classical phenomenon of volumetric locking. If attackers can directly use these known mechanisms to construct computationally expensive inputs, what additional value does introducing a complex multi-agent search framework provide? The paper does not sufficiently demonstrate that CloakFEM has clear advantages over methods that directly use existing numerical knowledge. Therefore, its practical contribution and application value remain unclear.

W3. The experimental comparisons are not sufficiently fair, making it difficult to demonstrate the practical advantages of the method.

In the main experiments, CloakFEM uses FEM domain knowledge and structured modification operations, while the conventional black-box baselines mainly search over encoded parameters. Therefore, their search capabilities are not entirely consistent. Although Section 4.2.2 uses a unified restricted search space, Table 2 reports median amplification factors of only 1.038× and 1.017× for CloakFEM and Evolutionary Search, respectively, indicating a limited difference in practice. In addition, the experiments only equalize the number of FEM solver calls without considering the additional costs of extensive LLM inference. Therefore, the current results are not sufficient to demonstrate that CloakFEM has significant practical advantages over conventional search methods.

W4. The paper has obvious formatting and figure issues, and the experimental organization and result presentation are also confusing, seriously affecting readability.

The paper contains several obvious formatting and labeling problems. For example, references on pages 2 and 3 are displayed incorrectly, some text in Figure 1 is incorrectly truncated, and names such as pdeagent_cd_3d_asymmetric_beta appear without explanation. The plots and legends in Figure 4 are also unclear. In addition, the experimental settings are inconsistent, with different experiments using different data subsets, search spaces, and numbers of generated candidates, making the results difficult to compare. The experimental section also reports many secondary statistical metrics without clearly highlighting the most important results or their relationship to the core contributions. These problems significantly affect the readability of the paper and the understandability of its experimental conclusions.

Questions for authors

See W2.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer DE4w

Thank you for the detailed review and for acknowledging the exploratory and engineering value of this work. We address your concerns about clarity and soundness below.

W1. Methodological contribution

CloakFEM provides a semantic search framework for heterogeneous FEM inputs under explicit validity and execution constraints. Its shared IR exposes physical inputs and supported node edits, source-specific writers reconstruct executable native inputs, and deterministic pre-execution checks and post-solve validation determine candidate acceptance. Solver methods, tolerances, execution limits, and resolution controls remain protected. The Strategy, Generation, and Selection components respectively identify edit targets, instantiate concrete magnitudes, and allocate solver queries using execution history. This design connects semantic reasoning to an auditable, solver-independent execution interface.

RQ3.1 directly quantifies the search gains from semantic generation, solver feedback, and learned selection (Figure 6(a, c); Appendix A.3). Replacing semantic Generation with a zero-knowledge generator reduces mean amplification from 127.38× to 10.25× and P90 from 188.13× to 1.46×. Removing solver feedback lowers these metrics to 46.00× and 42.09×, while removing learned Selection lowers them to 51.74× and 17.66×. These ablations hold the IR, validity policy, and query cap fixed, demonstrating substantial contributions from CloakFEM’s generation, feedback, and selection mechanisms on the evaluated cohort. We report both arithmetic mean and P90 to distinguish aggregate amplification from upper-tail behavior.

W2. Added value over known numerical mechanisms

Near-incompressibility, coefficient contrast, and mesh distortion provide useful numerical knowledge. The task addressed by CloakFEM is to turn such knowledge into effective, executable FEM program for red-teaming testing: identifying supported fields, coordinating modifications, choosing their magnitudes, and adapting to solver feedback while satisfying the acceptance constraints. A known difficult regime supplies a search direction; CloakFEM automates the seed-specific search needed to realize computational amplification.
As in computer-system red teaming, agents can turn established domain knowledge, such as buffer-overflow vulnerabilities, into concrete test inputs for effective vulnerability discovery. Similarly, CloakFEM can help FEM solver developers identify limitations in their implementations and use these findings to develop mitigation strategies.

Table 2(a) directly compares CloakFEM with a non-LLM domain-rule baseline equipped with search memory and coordinated parameter edits, serving as an explicit numerical-knowledge control. Built within the CLOAKFEM framework, the baseline follows a fixed schedule of domain-informed rules, modifying one or two parameters at predefined seed-relative magnitudes, and supports small sampled node movements. Its memory retains the best accepted input as the parent and records proposed candidates to avoid duplicates. Appendix A.1 details the baseline and bounded-edit protocol.

All three methods are evaluated on 444 PDE-Agent seeds. Table 2(a) reports results on the same 439 seeds with finite outcomes for every method, reproduced below for convenience.

MethodMeanMaxP90>3×
Evolutionary1.2818.0001.7632.1%
Non-LLM domain-rule1.4418.3332.4825.7%
CloakFEM (GPT)1.58512.8113.05210.5%

Against this specific Non-LLM domain-rule baseline, CloakFEM’s its paired rate above 3× is 4.78 percentage points higher, almost douled than the baseline(95% CI [2.73, 7.06]).

AuthorsOpenReview ↗

Reply to Reviewer DE4w (continued)

W3. Fair comparison and end-to-end search efficiency

Matched action space. As specified in Section 4.2.2, all three methods operate within the same bounded edit space and share the execution interface, validity checks, and query budget.

CloakFEM matters. Table 2(a) show a pronounced upper tail that is relevant to resource-exhaustion risk. In the comparison on 439 common finite seeds, CloakFEM achieves P90 amplification of 3.052× versus 1.763× for Evolutionary Search, and exceeds 3× on 10.5% of seeds versus 2.1%—approximately five times as frequently. Its maximum amplification is 12.811×, increasing linear iterations from 95 to 1,217. Thus, bounded, valid-input edits can still produce substantial computational amplification, and CloakFEM discovers these high-cost cases more reliably.Appendix A.1 provides paired statistical analysis and absolute iteration counts. WEDGE demonstrates the utility of performance-stressing inputs for exposing bottlenecks and improving code efficiency, PathoGen generates pathological inputs causing practical slowdowns. These studies motivate our emphasis on high-cost input discovery alongside median behavior.

Including LLM overhead. The equal-time comparison in Table 2(c) includes model calls and retries within the same total search deadline. CloakFEM retains higher mean amplification and a higher frequency of high-amplification inputs than evolutionary search after this overhead is included. Appendix A.1 details the protocol and outcome accounting.

W4. Case identifiers and experimental organization

We have checked the manuscript for the reported reference-display issue on pages 2-3 and would appreciate the specific citation or cross-reference location so that we can verify the exact item.
The identifier pdeagent_cd_3d_asymmetric_beta, which is a task ID, denotes a three-dimensional convection–diffusion case from PDE-Agent-Bench, with an asymmetric advection vector. Its source is YusanX/pde-agent-bench, data/benchmark_v2.jsonl, case cd_3d_asymmetric_beta; the corresponding repository IR file is docs/seed_corpus/ir_out/pdeagent/pdeagent_cd_3d_asymmetric_beta.json
We have updated Fig.2 and Fig.4 .

The experiments use cohorts selected for distinct evaluation questions and execution requirements. Under the experimental budget, this organization enables broad main evaluation and targeted controlled comparisons:

ExperimentCohortPurpose
Primary search (Table1, Fig 4, Table 4, Fig.4)1,027 seeds: 577 Elmer, 444 PDE-Agent, 6 NAFEMSAmplification prevalence and complete-system search effectiveness
Bounded-edit control (Table 2(a))444 PDE-Agent seedsComparison under matched actions
Component ablation (Fig. 6)444 PDE-Agent seedsContribution of the model components
Non-LLM domain-rule comparison (Table 2(a))444 PDE-Agent seedsAdded value over domain rules with incumbent memory and two-field edits
Cross-solver replay (Fig.5, Table 10, Fig.9-10)99 selected saved cases supported by the replay adaptersReproducibility of selected inputs across implementations

The 667-seed count-metric cohort is the finite intersection across all seven methods. Count and wall-time metrics are reported separately. Secondary distribution summaries are provided in the appendix.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns.

Regarding W4, the PDF reference issue is clearly observable. For example, the left border of page 3 and the header "Under review as a conference paper at ICLR 2027" are incorrectly linked to the reference by Gilbert Strang and George J. Fix, An Analysis of the Finite Element Method. Similar abnormal hyperlink regions also appear on page 2.

Overall, while the additional experiments have addressed some of my concerns, I do not think they sufficiently resolve my main concerns. Therefore, I will maintain my original score.

Thank you for pointing out the abnormal hyperlink regions on pages 2–3. We have fixed this in the revised PDF and verified that the page border and running header are no longer linked. We believe this issue comes from the PDF rendering instead of our reference mistake.

We also note that you find the main concerns insufficiently resolved. Could you clarify which concerns remain insufficiently addressed so that we can improve the revised manuscript?

Reviewer_vVHTOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CloakFEM combines multiple LLM roles, a structured input representation, and solver feedback to search for FEM inputs that pass validity checks and complete successfully while requiring more computation. The paper evaluates cost amplification across multiple solver stacks and physical problems, with bounded-edit controls, cross-solver replay, and ablations.

Critical strengths
  1. The combination of structured edits, deterministic validation, and actual solver feedback provides a reproducible framework for FEM stress testing.

  2. The evaluation covers diverse settings and includes bounded edits, cross-solver replay, and mitigation experiments, helping characterize when amplification occurs.

Critical weaknesses
  1. The dependence of large amplification effects on changes to the underlying task remains unclear. Edits can change both physical difficulty and computational size. For example, the annulus case increases iterations by 4.39×, but its node count also grows from 2,099 to 2,512. Although bounded-edit controls are provided, it remains unclear how much amplification persists at matched problem sizes. This distinction matters when assessing whether the method mainly constructs harder simulation tasks or exposes substantial computational sensitivity to constrained input changes.

  2. The comparisons do not sufficiently establish the added value of repeatedly using LLMs. The domain-rule baseline mainly scans individual parameters and lacks trajectory memory and coordinated multi-parameter edits, both available to CloakFEM. Some gains may therefore come from a more capable search process rather than LLM reasoning itself. A stronger non-LLM baseline with these capabilities would better isolate the contribution. Moreover, matching solver-query counts does not establish whether the advantage persists after accounting for model-call overhead.

Questions for authors
  1. Under the strict-relative editing constraints, what fraction of cases still achieves more than 3× cost amplification when the actual number of degrees of freedom is held fixed?
  2. Does CloakFEM retain a clear search advantage over a non-LLM domain-rule baseline equipped with trajectory memory and coordinated multi-parameter edits?
  3. Under an equal total search-time budget that includes model-call overhead, does CloakFEM still find inputs with greater cost amplification?
4 comments
AuthorsOpenReview ↗

Reply to Reviwerer vVHT

Thank you for the constructive review and for recognizing the value of our framework and evaluation. We address your concerns and clarify the relevant details below.

W1 & Q1. Amplification after normalizing for DOF growth

To quantify the amplification remaining after accounting for problem-size growth, we retrospectively normalize the completed strict-relative results. We reconstruct baseline and winner DOFs using the archived mesh generator and the finite-element spaces used in the experiments. With N denoting the number of scalar unknowns before boundary-condition elimination and C the solver-cost metric for each seed, we apply the approximate linear size correction ρ_norm = [C(x⋆)/C(x₀)] / [N(x⋆)/N(x₀)].

MethodMeanP90>1.5×>3×
Evolutionary1.2341.65513.4%1.8%
CloakFEM (GPT)1.5162.88026.2%8.4%

We updated Table2(b) show that CloakFEM retains a substantial search advantage after a linear correction for DOF growth.

W2 & Q2. Non-LLM domain-rule baseline

There is no off-the-shelf non-LLM baseline for this setting that combines domain-informed edits, search memory, and joint parameter changes, so we implemented a non-LLM domain-rule baseline within the CLOAKFEM framework, retaining search memory and support for joint parameter edits while replacing LLM-generated proposals with fixed domain-informed rules. The baseline retains the best accepted input as the parent of subsequent proposals and records proposed candidates to avoid duplicates. It follows a fixed rule schedule, modifying one or two parameters at predefined seed-relative magnitudes, and supports small sampled node movements. Rule order and magnitudes remain fixed.

The completed comparison uses 444 PDE-Agent seeds. All three rows below use the same 439 seeds with finite outcomes for every method (Table 2(a)).

MethodMeanMaxP90>3×
Evolutionary1.2818.0001.7632.1%
Non-LLM domain-rule1.4418.3332.4825.7%
CloakFEM (GPT)1.58512.8113.05210.5%

Under the same bounded action space and budget, CloakFEM thus achieves higher average amplification and threshold rates than this specific rule policy with incumbent memory and two-field templates.

Q3. Equal total search time including model calls

The completed experiment assigns each method a 1,800-second budget and removes the solver-query cap. The parent-process timer covers worker startup, imports, baseline execution, proposal/model calls, retries, validation, and checkpoint writes. Sec 4.2.2 describes setting details. The same solver configuration and seed-relative input constraints apply to both arms. The results are shown in Table 2(c).

MethodMeanP90>3×
Evolutionary1.4622.6566.4%
CloakFEM (GPT)1.5933.14812.8%

These summaries and Table 2(c) use the same 437 shared finite seeds.

Dear Reviewer vVHT,

Thank you for your thoughtful feedback and for highlighting the reproducible framework and the breadth of our evaluation.

In our rebuttal, we addressed your questions about problem size, the added value of LLM-guided search, and model-call overhead. We added an approximate degree-of-freedom adjustment, a domain-rule baseline with search memory and coordinated edits, and an equal-time comparison that includes model-call time. After the size adjustment, CloakFEM finds more cases above 3× amplification than evolutionary search (8.4% vs. 1.8%). It also outperforms the domain-rule baseline (10.5% vs. 5.7%) and evolutionary search under the equal-time budget (12.8% vs. 6.4%) on this measure. These results support CloakFEM’s advantage under the conditions you raised.

With about one day remaining in the author–reviewer discussion period, we would appreciate it if you could review our response and let us know whether it addresses your concerns. If any questions remain, we would be happy to discuss them before the deadline.

Best regards, The authors of CLOAKFEM: Agentic FEM Code Generation for Red-Teaming Computational Resource Exhaustion

I appreciate the authors' thoughtful response, which has helped address some of my concerns. I will increase my score to 3.

Thank you for taking the time to reassess our work, and we appreciate your recognition of our efforts.

Reviewer_RvzsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper uses an LLM pipeline to find FEM inputs that take more computation, while the authors want these inputs to remain valid and convergent without changing the solver configuration. The pipeline edits a common IR and leaves execution and validity checking to ordinary code. The authors report results on 1,068 seeds, compare with generic search methods, and examine bounded edits, transfer to other implementations, and defenses.

Critical strengths
  1. I like the problem. A FEM input is very much of a physical description, while its computation also depends on how the description meets the discretization and solver, so validity does not buy an easy solve. I feel like systematically searching for this gap can make an interesting paper. I would count a careful FEM investigation as enough of a contribution, even if the three agents mostly assemble existing ideas.

  2. I can see a real amount of engineering here. The authors support several FEM execution paths and put together 1,068 seeds. I especially like the bounded edits and transfer experiments, as a large unrestricted edit could make the problem very different, while reproducing an attack in another implementation helps explain whether the difficulty belongs to one code path. These experiments are a large part of why I was leaning to accept after reading the paper. What bothers me is that some of the code does a different experiment.

Critical weaknesses
  1. I have a problem with the success meaning here. In cloakfem/solvers/elmer_native_solver.py, the adapter looks for ALL DONE, return code zero, and the absence of selected fatal messages. A constructed log with a linear non-convergence message passes this gate. I should be careful with this example, as Elmer normally aborts when linear iterations reach the limit, so the example does not show that default linear solves have this problem. It applies when the seed permits continuation after linear non-convergence. There is also a nonlinear path: in the upstream code I checked, ComputeChange only aborts on nonlinear non-convergence if Nonlinear System Abort Not Converged is enabled. Therefore, I do not get a convergence guarantee from normal program termination for all these cases. The attacker does not need to change the protected abort option if the seed already allows this behavior. Since the package does not pin the Elmer source version or contain the original native snapshots, I would look at the actual configurations and winner logs before deciding how many results have this problem.

    For materials, I find that cloakfem/sif_edit.py skips an expression it cannot parse as a number, while the later check still reads the old seed material. Changing heat conductivity from 1.0 to Real $ -1.0 passes the precheck and, with a constructed successful solve result, the postcheck as well. This does not tell me that an actual winner has negative conductivity. It tells me that the gate can give its answer for the old material while Elmer receives the new one, so checking the actual winners is still necessary before I call them valid expensive solves.

  2. I also find problems in the experiments I initially trusted most.

    In cloakfem/irv2_constraints.py, _is_poisson_path treats every nu as Poisson's ratio, including fluid viscosity. Calling StrictRelativeGate(ir).failures(ir) on the 444 submitted PDE-Agent seeds rejects 40 unchanged inputs, and 28 have no feasible candidate because the factor-of-two lower bound is already above the imposed 0.49 cap. Keeping the original input at amplification one does not make it pass these constraints. I do not know whether correcting this mistake makes the LLM advantage larger or smaller, as both methods may suffer from it. However, the current statistics describe a different feasible space, so I would recompute Table 2 before using it to judge the bounded-edit advantage.

    I first suspected that the main baselines had no node edits, as run_baselines.sh uses native actions. What I overlooked is that aggregate_rq1_irv2_comparison.py reads rq1v2_irv2/, and cloakfem/llm/irv2_baselines.py does have node channels. Appendix A.1 also already says the unrestricted comparison is not fully action-space matched. Therefore, I withdraw the no-node interpretation. I guess the native script may be an old entrance left in the package; I only need the actual commands and records behind Table 5 to settle this part.

    For transfer, I do not get the exact winner replay described in Section 4.3. assemble_canonical_elasticity_skfem in cloakfem/solvers/cross_solver.py replaces the saved boundary conditions and source with zero displacement and fixed body forcing. A real scikit-fem assembly probe changes the source and Dirichlet values by large amounts, but the matrix and right-hand side remain identical. The solver factory also reuses the seed mesh for some candidate geometry changes. I can believe the material effect survives another assembly, while I cannot use this experiment to say the complete attack transfers. Similarly, seeing the effect in several displacement formulations does not tell me it is "problem-intrinsic volumetric locking."

    The timeout accounting also looks kind of suspicious once I check _run_seed_killable, as it waits for the child to exit before draining its queue. In a probe, the worker finishes computing and returns a synthetic 1 MB result, but the wrapper kills it as timed out. A checkpoint may still contain the best amplification. Therefore, I would first check the completion and timing statistics, rather than assume the saved ratios are wrong, since waiting to return an answer is a different time cost from searching.

    I would also discuss WEDGE and PathoGen. They already use LLM assistance for performance-stressing or pathological inputs. I think the FEM semantics and solver-blind interface can still be a useful difference, which the authors should explain against these methods. Missing these references is not what changes my rating here.

I wanted to accept this paper, since I like the question and can see enough engineering for a serious study. I do not think asking for a completely new agent architecture would help much. However, most of the contribution comes from the experiments, and I need their numbers to be worth-trusting. The code currently leaves me unsure whether a successful input really meets the validity conditions, and whether a replay really uses that input. I would change my rating if the authors connect the tables to the actual runs and correct the affected results, while a code patch alone does not tell me what happens to the old numbers.

Questions for authors
  1. Does Table 5 use the IR-v2 baselines with node edits? Please give the commands and corresponding records. For Table 2, how do the 28 seeds with no feasible candidate enter the statistics, and does the advantage remain after correcting the constraints?
  2. Among the 618 Elmer seeds, which allow linear non-convergence to continue, and how do the authors verify final nonlinear convergence? I would like the affected winner logs and a check of the actual material values, with any resulting changes to the reported results.
  3. Which transfer results preserve the saved boundary conditions, source, and geometry, and which reconstruct a canonical problem? I would also like to know whether the queue problem changes the completion and timing results.
5 comments
AuthorsOpenReview ↗

Reply to Reviewer Rvzs

Thank you for the thoughtful review and for recognizing the value of the problem, the engineering effort, and the bounded-edit and transfer experiments. We appreciate your careful examination of the code and address your concerns below.

1. Q1&W1: IR-v2 baselines and reproduction commands of Table 5

Table 5 uses the IR-v2 baselines with node-edit channels. The earlier package exposed the legacy native-action launcher as run_baselines.sh. The table’s actual aggregation reads experiments/results/rq1v2_irv2/. We have cleaned our codebase and make a more clear reproduction instructions explicitly identify the IR-v2 launcher:

for m in random evolutionary cmaes oneplusone ngcma; do
  bash experiments/run_baseline_irv2_one.sh "$m" 22
done
python experiments/aggregate_rq1_irv2_comparison.py

The retained driver logs confirm 22 shards per method. The launcher sets --action-space irv2 --budget 40 --rep-seed 0 and uses manifest_nosimjeb_t1800.jsonl. The historical campaigns retained 1,068 records per method. Subsequent baseline review excluded another 41 Elmer cases, leaving the current 1,027-case evaluation dataset. The launcher now defaults to evaluation_manifest.jsonl. Table 5 uses the seven-method finite intersection of 667 count-based and 355 wall-time cases.

2. Q1&W2: physics-aware bounds and recomputed results of Table 2

We have the gate to distinguish fluid viscosity from an elastic Poisson ratio when a field is named nu. The physics-aware correction changes the feasible intervals of 43 fluid seeds, including the 28 cases with an empty feasible space. We reran both methods on all 43 and retained 401 unaffected records per method. With the same corrected gate, at most 40 queries, and a 1,800-second limit, the results changed to:

MethodMeanMedian> 1.5×> 3×
OriginalCorrectedOriginalCorrectedOriginalCorrectedOriginalCorrected
Evolutionary1.2221.2811.0171.05114.81%17.77%1.37%2.05%
CloakFEM (GPT)1.5551.5851.0381.09225.24%27.79%10.68%10.48%

These results show that CloakFEM retains an advantage under the corrected constraints.
Table 2(a) reports the final bounded-edit(strict-relative) comparison, with paired effect estimates and sensitivity analysis in Appendix A.1.

3. Q2&W1: Native Elmer: non-convergence seeds

We have reviewed all 618 native Elmer baselines. Of these, 577 satisfy the applicable numerical criteria: 568 numerical tasks and nine completed noniterative workflows. The remaining 41 cases are excluded from datasets(41 excluded task IDs). No nonconverged cases were detected in the checked source-side runs for PDE-Agent or NAFEMS, whose cohorts remain unchanged.

The full retained native log and trace archive is approximately 11.88 GB, which is too large for this repository upload. So we provided a lightweight Elmer log package containing logs, native SIF inputs, and audit receipts for five illustrative cases that allow readers to inspect the native output and supporting evidence. and. We used Elmer 26.2-devel.

Our treatment of winners is different because CLOAKFEM is an adversarial method designed to exhaust computational resources. The attack does not require eventual convergence because the attacker’s goal is to keep the server busy and delay other users’ requests, rather than obtain a computation result.

On the 222 native Elmer count cases shared by all seven methods after baseline filtering and coefficient screening, CloakFEM(GPT) achieves median amplification of 5.00× versus 1.17× for evolutionary search; 64.4% versus 17.6% exceed 3× (Table 6(b)).

4. Q2&W2: Effective material values

We audited all seven methods’ saved native Elmer winners. Within the revised 577-case cohort, the initial screen identified 34 introduced coefficient-bound violations and 21 unresolved winner records, spanning 24 distinct seeds. We fixed the shared validator to evaluate the actual SIF supplied to Elmer, including MATC expressions, tensor coefficients, and mesh-dependent ranges and reran all seven methods on all 24 affected seeds under the corrected gate. All 168 selected endpoints passed independent execution and material revalidation. We retained the screened historical results for the other 553 seeds.

The advantage persists after correcting and rerunning the affected cases. Figure 4 has been updated accordingly to reflect the corrected amplification distributions, query-prefix progress, and similarity–amplification tradeoffs.

AuthorsOpenReview ↗

Reply to Revierwer Rvzs (continued)

5. Q3&W2: Transfer

The original cross-solver replay adapters had compatibility limitations when translating saved inputs between solver backends. The canonical elasticity replay substituted source terms and boundary conditions, and some geometry edits could reuse the seed mesh. We corrected the adapters and replaced the earlier evidence with full saved-input replay, reported in revised Section 4.3 and Appendix A.2.

The replay freezes the original saved winners without target-specific search. Per-case records link each winner to its original search record, saved input, generated mesh, and backend results. For the 99 retained pairs, all backend–input–mesh identity checks pass. Separate assembly checks verify that source and nonzero Dirichlet changes affect the assembled right-hand side, material changes affect the stiffness matrix, and geometry and node edits reach the generated mesh.

Correcting the replay strengthens the observed transfer results: the number of Stokes pairs exceeding 1.5× amplification on both assembly targets increases from 17/34 to 34/34, and convection–diffusion increases from 4/8 to 8/8, with all these corrected pairs converging on both targets. Overall, 97/99 retained pairs exceed 1.5× on both targets. Table 10 distinguishes 88 pairs with converged amplification from nine additional pairs with above-threshold capped-work lower bounds. Figure 5 has been updated to compare source and target amplification; 34 elasticity pairs converge and preserve amplification within a factor of two on both targets.

We defined “problem-intrinsic volumetric locking” as a shared problem/formulation-level weakness to distinguish from an implementation-specific weakness. Our objective is to generate valid FEM programs that expose common computational weaknesses across solvers. In the near-incompressible elasticity regime associated with volumetric locking, the same saved inputs reproduce source-scale amplification in 34 converged pairs on both assembly targets, while 43/45 pairs exceed 1.5× with convergence in native Elmer. These results support the intended distinction: the expensive behavior persists beyond a single solver’s code path. Revised Section 4.3 and Appendix I make this scope explicit and disclose the shared Krylov driver; native Elmer provides complementary evidence from a separate solver implementation.

6. W2: Completion transport

In five controlled replays of the original Queue-based wrapper using a retained 10.3 KB result, the median overhead from computation completion to result return was 0.00296 seconds, corresponding to only 0.000164% of the 1,800-second budget. These measurements indicate that result transport has little effect on overall timing under normal return conditions. All methods within each comparison also used the same subprocess wrapper and deadline policy. Nevertheless, to make the implementation more rigorous, we revised the shared runtime to separate computation completion from result delivery and cleanup, ensuring that blocked result transport cannot cause a completed search to be misclassified as a search timeout.

The corrected runtime uses a compact completion status and an atomic result file, separates search, delivery, and cleanup timing, and enforces one search deadline. All five regression checks pass, covering a 1 MB result, checkpoint recovery at a real timeout, delayed delivery, crash classification, and rejection of stale files.

We retained 587 verified repaired-runtime records and reran the remaining 301 method–seed attempts. All 888 final records (444 per method) now use atomic-result-v2; frozen code and seed hashes and equality between per-seed and aggregate records are verified. Neither method recorded a result-delivery failure. Finite checkpoint outcomes at the search deadline are recorded separately from normal search returns.

The recomputed comparison continues to support a search advantage. The revised results appear in manuscript Table 2(c), with the deadline and outcome-accounting protocol in Appendix A.1. Detailed completion and timing records are retained in the accompanying evidence.

7. W2: Positioning relative to WEDGE and PathoGen

WEDGE synthesizes performance-characterizing constraints and uses instrumentation with coverage-guided fuzzing; PathoGen integrates LLM candidate generation into feedback-driven evolutionary search. We have discussed these two work in related work.

Dear Reviewer Rvzs,

Thank you for your thoughtful feedback and for highlighting the value of the research question and the engineering behind the bounded-edit and transfer experiments.

In our rebuttal, we addressed your concerns about the validity and reproducibility of the results by correcting the input constraints and material checks and rerunning the affected experiments. All 168 rerun endpoints passed independent material revalidation, and CloakFEM retained its advantage over evolutionary search under the corrected bounded-edit constraints (mean amplification: 1.585× vs. 1.281×). We also replaced the transfer analysis with full saved-input replay: 88 of 99 retained pairs exceeded 1.5× on both target implementations with convergence. We clarified convergence and timing accounting so that these outcomes are interpreted separately.

With about one day remaining in the author–reviewer discussion period, we would appreciate it if you could review our response and let us know whether it addresses your concerns. If any questions remain, we would be happy to discuss them before the deadline.

Best regards,

The authors of CLOAKFEM: Agentic FEM Code Generation for Red-Teaming Computational Resource Exhaustion

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.