- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes an adversarial arena for generating and evaluating finite-element computation problems. A single model acts as both generator and solver. The generator writes a numerical multiple-choice question, its answer, three distractors, and scripts computing each option. The arena reruns these scripts before testing the solver, first without tools and then with FEniCSx. A question is counted as solved only if the solver answers correctly under all four cyclic option orderings. The authors report an 866-round match using composer-2.5. Of 666 questions passing the script checks, 401 defeated the solver. Over the match, questions became longer, more coupled, and increasingly similar to their predecessors. The paper also collects 180 questions answered correctly in all four tool-enabled presentations, then evaluates the same model without FEM libraries. Reported multiple-choice accuracy is 32.2%; a separate, heuristic analysis estimates that 16.1% reproduce the reference numerical value.
Critical strengths
-
The arena makes a useful distinction between occasional success and consistent success. Requiring correct answers across multiple presentations exposes failures that a single successful attempt would hide. The separation between no-tool and tool-enabled solving also helps reveal when distractor structure allows the model to bypass the intended computation. These are consequential design choices for evaluating agents used in scientific workflows.
-
The analysis openly examines how the benchmark can go wrong. Sections 5.2–5.6 report near-duplicate generation, shrinking option gaps, missing answers, memory failures, and an evaluation-environment defect. This makes the paper more useful than a headline accuracy result alone. In particular, the observation that an adversarial generator can make problems harder largely by extending their procedural specifications is worth understanding.
Critical weaknesses
-
The verification checks do not establish that the reference answer correctly solves the stated problem. In Section 3.1, the arena checks that the generator’s scripts execute and reproduce the numbers the generator declared. That establishes consistency between those scripts and those numbers. It does not check that gt.py implements the equations, boundary conditions, discretization, and functional described in the question, or that each distractor makes only the intended mistake. This matters because “verifiably correct” questions are central to the contribution. Section 3.3 further describes the reference value as independently reproduced five times, but the four solver successes are correct multiple-choice selections; the described protocol does not require their computed values to match the reference. Section 5.6 itself shows why that distinction matters. I am not claiming that the references are wrong. I am saying that the evidence presented does not yet justify the correctness guarantee, especially when generator and solver use the same model and library.
-
The experiments do not yet cleanly establish difficulty calibration or the size of the library-removal effect. The generator is rewarded for any solver failure, rather than for finding questions near a particular success threshold. Its late-stage behavior—near-duplicates, longer specifications, and a 98% win rate—could represent exploitation of the contract rather than calibration to a useful ability boundary. Without a comparison against generation without solver feedback, it is difficult to tell what the adaptive loop adds. The FEM-free experiment has a related interpretation problem. Its 100% library-enabled reference performance is guaranteed by selection of the bank, rather than measured in a fresh evaluation. Moreover, 123 of the 180 FEM-free sessions ran with a broken interpreter setup (Section 5.6, lines 413–421). Removing flagged sessions does not remove all sessions exposed to that defect. The statement that this makes the result “conservative” needs care: an artificially lower FEM-free accuracy would exaggerate the apparent performance drop. Finally, the 16.1% numerical-reproduction result relies on an extraction heuristic that has not been checked against human labels. The results suggest substantial difficulty, but the headline comparison needs a cleaner experiment.
Questions for authors
-
Was any sample independently checked for agreement between the question text and the reference implementation, beyond rerunning the generator’s scripts? Please describe the sampling procedure and any disagreements found. For the collected bank, do the four successful tool-solver runs actually reproduce the reference numbers, or only select the correct options? An independent audit would substantially affect my confidence in the benchmark.
-
Can you provide a fresh, paired evaluation of the bank with and without FEniCSx, using the corrected environment and matched option arrangements, scoring rules, and resource budgets? Please also validate the numerical-value extraction against manually checked transcripts. This would establish how much of the reported gap survives the environment defect and success-based bank selection.
-
Is there a comparison with a generator that receives no solver-outcome history? A comparison accounting for question length, coupling, and problem-family duplication would help determine whether feedback finds additional useful difficulty. Positive evidence here would strengthen the central self-play contribution.
-
How does accuracy across four different option orderings compare with four independent calls using the same ordering? Also, how is the reported 71% per-attempt accuracy affected by cancelling unfinished siblings after the first failure? These answers would help distinguish sensitivity to ordering from ordinary run-to-run variability and completion-dependent sampling.