acceptodds
Under review as a conference paper at ICLR 2027

Adversarial Self-Play for Scientific-Computing Agents: An Arena Where a Model Writes Finite-Element Problems It Cannot Solve

Abstract

Static benchmarks for tool-using scientific agents saturate quickly and are vulnerable to contamination, while human-authored problems are expensive and rarely calibrated to the frontier of what an agent can do. We introduce an adversarial, zero-sum arena in which a single language-model agent plays both sides of a game about numerical finite-element computation: a Generator designs a multiple-choice question, together with a ground-truth value, three distractors, and four verification scripts that independently reproduce every option with FEniCSx on a fixed, tagged mesh; a Solver, the same model, must answer the question, first without tools and then with a sandboxed Python/FEniCSx environment. The arena executes the generator's scripts, forfeits any round whose question is not fully verified, presents each question under four cyclic option orderings, and requires the solver to be correct on all four. Every invocation runs under fail-closed filesystem, CPU, and memory isolation, and every call is metered for time, tokens, and cost. We report a match of rounds played by a single frontier coding model. Of the verified questions, the solver answered and failed , despite a per-attempt tool-solver accuracy of . The generator's strategy visibly evolves: median question length grows from k to k characters and questions progress from single PDEs to chains of six to eight coupled sub-problems; the no-tool solver, meanwhile, exploits structure in the distractors to answer of verified questions without computation. We analyze these dynamics, the forfeit rate caused by the generator's own verification scripts, and the cost profile of the match ($1.4k, 5.9B tokens). The arena yields a self-refreshing, verifiably correct, and difficulty-calibrated question bank, and a companion evaluation mode replays that bank against agents that must implement the finite-element method themselves. Code, prompts, and the full match log are released.

Primary area:
datasets and benchmarks
Keywords:
benchmark generation, LLM agents, tool use, adversarial self-play, evaluation, scientific machine learning, finite element method, execution-based verification, sandboxing, data contamination

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PiBbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes an adversarial arena for generating and evaluating finite-element computation problems. A single model acts as both generator and solver. The generator writes a numerical multiple-choice question, its answer, three distractors, and scripts computing each option. The arena reruns these scripts before testing the solver, first without tools and then with FEniCSx. A question is counted as solved only if the solver answers correctly under all four cyclic option orderings. The authors report an 866-round match using composer-2.5. Of 666 questions passing the script checks, 401 defeated the solver. Over the match, questions became longer, more coupled, and increasingly similar to their predecessors. The paper also collects 180 questions answered correctly in all four tool-enabled presentations, then evaluates the same model without FEM libraries. Reported multiple-choice accuracy is 32.2%; a separate, heuristic analysis estimates that 16.1% reproduce the reference numerical value.

Critical strengths
  1. The arena makes a useful distinction between occasional success and consistent success. Requiring correct answers across multiple presentations exposes failures that a single successful attempt would hide. The separation between no-tool and tool-enabled solving also helps reveal when distractor structure allows the model to bypass the intended computation. These are consequential design choices for evaluating agents used in scientific workflows.

  2. The analysis openly examines how the benchmark can go wrong. Sections 5.2–5.6 report near-duplicate generation, shrinking option gaps, missing answers, memory failures, and an evaluation-environment defect. This makes the paper more useful than a headline accuracy result alone. In particular, the observation that an adversarial generator can make problems harder largely by extending their procedural specifications is worth understanding.

Critical weaknesses
  1. The verification checks do not establish that the reference answer correctly solves the stated problem. In Section 3.1, the arena checks that the generator’s scripts execute and reproduce the numbers the generator declared. That establishes consistency between those scripts and those numbers. It does not check that gt.py implements the equations, boundary conditions, discretization, and functional described in the question, or that each distractor makes only the intended mistake. This matters because “verifiably correct” questions are central to the contribution. Section 3.3 further describes the reference value as independently reproduced five times, but the four solver successes are correct multiple-choice selections; the described protocol does not require their computed values to match the reference. Section 5.6 itself shows why that distinction matters. I am not claiming that the references are wrong. I am saying that the evidence presented does not yet justify the correctness guarantee, especially when generator and solver use the same model and library.

  2. The experiments do not yet cleanly establish difficulty calibration or the size of the library-removal effect. The generator is rewarded for any solver failure, rather than for finding questions near a particular success threshold. Its late-stage behavior—near-duplicates, longer specifications, and a 98% win rate—could represent exploitation of the contract rather than calibration to a useful ability boundary. Without a comparison against generation without solver feedback, it is difficult to tell what the adaptive loop adds. The FEM-free experiment has a related interpretation problem. Its 100% library-enabled reference performance is guaranteed by selection of the bank, rather than measured in a fresh evaluation. Moreover, 123 of the 180 FEM-free sessions ran with a broken interpreter setup (Section 5.6, lines 413–421). Removing flagged sessions does not remove all sessions exposed to that defect. The statement that this makes the result “conservative” needs care: an artificially lower FEM-free accuracy would exaggerate the apparent performance drop. Finally, the 16.1% numerical-reproduction result relies on an extraction heuristic that has not been checked against human labels. The results suggest substantial difficulty, but the headline comparison needs a cleaner experiment.

Questions for authors
  1. Was any sample independently checked for agreement between the question text and the reference implementation, beyond rerunning the generator’s scripts? Please describe the sampling procedure and any disagreements found. For the collected bank, do the four successful tool-solver runs actually reproduce the reference numbers, or only select the correct options? An independent audit would substantially affect my confidence in the benchmark.

  2. Can you provide a fresh, paired evaluation of the bank with and without FEniCSx, using the corrected environment and matched option arrangements, scoring rules, and resource budgets? Please also validate the numerical-value extraction against manually checked transcripts. This would establish how much of the reported gap survives the environment defect and success-based bank selection.

  3. Is there a comparison with a generator that receives no solver-outcome history? A comparison accounting for question length, coupling, and problem-family duplication would help determine whether feedback finds additional useful difficulty. Positive evidence here would strengthen the central self-play contribution.

  4. How does accuracy across four different option orderings compare with four independent calls using the same ordering? Also, how is the reported 71% per-attempt accuracy affected by cancelling unfinished siblings after the first failure? These answers would help distinguish sensitivity to ordering from ordinary run-to-run variability and completion-dependent sampling.

Reviewer_vhmbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposed a role-play-based adversarial method for one model to iteratively propose and solve finite-element problems, with and without the tools. Thus, the paper wants to use this method to iteratively build a good set of problem lists of suitable difficulty while the model's ability to use the tool sets properly can be evaluated.

Critical strengths

It is indeed a proper question to ask whether a model learns to reason and solve a finite-element problem or simply get a number.

The paper also proposed a concrete, inspectable scientific-agent evaluation artifact. The fixed mesh, controlled execution environment, archived question/reference/distractor scripts, full prompts, and explicit treatment of forfeits make the proposed arena substantially more reproducible than unrestricted question generation. Reporting both presentation-level and all-four-gate outcomes exposes an operational reliability dimension that ordinary answer accuracy would miss

Critical weaknesses
  1. Benchmark validity is weaker than the central correctness/calibration claims. The verifier establishes that four model-written scripts execute and reproduce their declared values; it does not independently check whether the reference implements the stated PDE, boundary conditions, discretization, quadrature, and rounding, or whether each distractor represents the claimed mistake (Section 3.1). Four correct tool-solver letters also do not necessarily mean four independent numerical reproductions (Section 3.3), as the FEM-free analysis itself illustrates. The reward favors defeating the solver, without an explicit target difficulty or diversity constraint; the late trajectory reaches 98% failures and character similarity 0.998. Thus a valid self-refreshing bank cannot be inferred from executable consistency and increasing failure rate alone. An independent semantic/numerical audit is necessary, particularly for coupled and closely spaced-option questions.

  2. The key empirical interpretations lack controlled comparisons. The tool baseline is 100% because those successes select the bank, rather than because fresh tool-enabled evaluation achieves 100%. Moreover, 123/180 FEM-free sessions had an interpreter/NumPy defect, and the corrected 57 sessions cover different, harder questions (Section 5.6). Calling this “conservative” does not establish the size of a library-removal effect: impairing the comparison environment can exaggerate that gap. Exact reproduction is additionally measured by an unvalidated extraction heuristic. Finally, there is no no-history or length-matched generation baseline to show that reward feedback, rather than the explicit deepening prompt and longer specifications, produces the reported escalation. These limitations affect the principal claims, beyond ordinary requests for more models.

Questions for authors
  1. Can an independent FEM implementation or blinded expert audit check a stratified sample of reference–question pairs, including long coupled questions and the smallest option gaps? Please report semantic mismatch/ambiguity rates separately from script-execution failures. Do the stored tool transcripts substantiate numerical reproduction, rather than only correct letters?

  2. Can the bank be replayed under the repaired environment in both tool-enabled and FEM-free conditions, with fresh calls, matched answer/scoring protocols, and clearly accounted budgets? Manually validating the 29 exact-reproduction classifications is also needed for that headline. This would directly address selection effects and the environment defect.

  3. What happens with outcome history removed or shuffled, while retaining the same deepening/exploring prompt and a matched length/compute budget? If this comparison is unavailable, please restrict “reward-driven” and “difficulty-calibrated” conclusions to descriptive observations of this trajectory.

  4. How are 356 cumulative agent-hours compatible with 15.6 active match-hours and four gate slots? Please also specify block-bootstrap block length/replicates and explain how cancelled presentations affect the 71% attempt-accuracy denominator and the no-tool histogram.

Reviewer_UXfiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a feedback loop for LLMs to create challenging problems that break itself.

Critical strengths
  1. The research problem authors identify (self-identifying weaknesses of an LLM itself) is important.

  2. Agent weaknesses are observed through experiments.

Critical weaknesses
  1. Paper only tests the methodology on one LLM, none of the conclusion/analysis is shown to apply across different LLMs.

  2. Generated problems are also not tested on other LLMs.

  3. More analysis onthe effect of feedback would be helpful to understand the self-challenging mehcanism. E.g. does feedback discover new failure modes or mostly just extend an existing failure mode?

Questions for authors
  1. Authors could discuss and compare their work with MathDuels [1], which also proposes generator/solver structure as a benchmark.

[1] MathDuels: a self-play benchmark that grows. Xu et al. COLM 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.