acceptodds
Under review as a conference paper at ICLR 2027

Reasoning over Simulations: Symbolic Intermediaries for LLM Design Agents with a Numerical Oracle

Abstract

Learning to improve a mechanism from simulation requires connecting observed motion to changes in its design. For language agents, this is a representation problem: simulated trajectories provide numerical evidence, while design revisions are proposed through linguistic reasoning. We introduce symbolic intermediaries, compact analytical expressions fitted to simulated trajectories and supplied alongside the target description. They expose geometric structure that agents can compare, discuss, and use to formulate revisions. A design, simulation, critique, and memory loop accumulates this experience at inference time while the language model's parameters remain fixed. Our analysis characterizes the information needed for local revisions, conditions for their transfer, and boundaries of the feedback representation. We evaluate the approach on MSynth, covering six planar target-curve families and three language models. Critique supports sustained refinement across models, while symbolic feedback provides an additional benefit that depends on the model and target motion. With critique and memory matched, the share of runs improving after the first third of search rises from to , and the median share of total improvement achieved in that later phase rises from to . Critique transcripts show agents using the fitted expressions in geometric comparisons and calculations. These results connect inference-time learning to the representation of simulation experience, where analytical descriptions can help language agents use observed mechanism behaviour to guide subsequent design decisions.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
LLM agents, symbolic regression, neurosymbolic reasoning, iterative self-refinement, feedback representation, physics-based simulation feedback, engineering design automation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YDykOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes adding a symbolic regression step (PySR) between a kinematic simulator (pylinkage) and an LLM agent for 2D planar linkage synthesis, fitting simulated 2D trajectory coordinates into explicit equations y=S^(x)y = \hat{S}(x) to assist iterative critique and revision. However, this manuscript is in a severely broken, unproofread, and unfinished state, padded to 50 pages with full-page trivial plots and unverified AI-generated text/proofs, while suffering from fundamental mathematical and statistical flaws. I strongly urge the Area Chair (AC) to Desk Reject (or Strong Reject) this submission.

Critical strengths

The high-level intuition of translating raw numerical trajectory coordinates into compact symbolic expressions for LLM critique is conceptually reasonable.

Critical weaknesses
  1. Severely Broken and Unfinished Manuscript (Strong Call for Desk Reject) The authors clearly did not perform even a basic compilation check or human proofreading before submitting to ICLR (notably, Appendix A discloses the use of generative AI to draft manuscript text and proof sketches, and the paper is littered with unverified build artifacts). Specifically:
  • Unresolved LaTeX references (?? and (?)) throughout the paper:
    • Page 17, Line 906: "value-preserving models (??)."
    • Page 17, Line 909: "through optimization oracles (??)."
    • Page 20, Line 1030: "optimisation is deliberate (??):"
    • Page 36, Line 1938: "iterative closest point algorithm (Drakoulis, 2023; ?)"
    • Page 43, Line 2308: "It departs from meta-learning (?) in having..."
  • Missing Table referenced as "Table ??":
    • Page 34, Line 1827: "A naive aggregate reading of Table ?? suggests SR-absent critiques are more analytical..." -- Not only is the reference broken (Table ??), the referenced taxonomy table itself is completely missing from the manuscript.
  • Truncated, half-written sentences:
    • Page 28, Line 1480: "Formally, one common formulation is 6." (sentence abruptly ends here).
    • Page 28, Line 1502: "Each shape is defined on 7." (sentence abruptly ends here).
  • Corrupted Bibliography and garbled in-text citations:
    • Page 10, Lines 487-494: The first two bibliography entries completely lack author fields, causing the main text to render nonsensical citation keys: "(son, 2024)" on Page 3, Line 166 and "(nob, 2022)" on Page 9, Line 469.
    • Page 11, Line 570: Author name rendered in all-lowercase as "brian ichter".
  • Chaotic, duplicate, and skipped Figure numbering & blatant page padding:
    • Duplicate Figure 5: Page 25 defines Figure 5 (the neuro-symbolic loop diagram), and Page 35 defines a second Figure 5 (Ground truth Ellipse).
    • Out-of-order and skipped numbers: Figure 8 appears on Page 33 before Figure 6 (Page 36) and Figure 7 (Page 37). Figures 17-20 do not exist at all -- the numbering jumps straight from Figure 16 (Page 46) to Figure 21 (Page 46).
    • Page padding: Pages 35-46 (12 full pages) are padded by placing single, elementary 2D matplotlib curves (one circle, one line, one ellipse per page) at huge scale to inflate the submission to 50 pages.
  1. Self-Contradictory Core Formulation (Branch Collapse on Closed Curves) Four of the six target curve families in MSynth (Circle, Ellipse, Lemniscate of Bernoulli, and NACA Airfoil) -- as well as generic four-bar coupler curves -- are closed, multi-valued curves where a single x coordinate corresponds to two or more y values. Yet the pipeline fits single-valued explicit functions y = f(x) via least squares (Page 4, Eq. 2; Page 31, Line 1672). Ironically, the authors themselves prove in Proposition B.5 (Page 22, Lines 1144-1155) that fitting y = f(x) to a symmetric closed curve like an ellipse collapses the upper and lower branches into the zero constant f(x) = 0, losing all shape information! And on Page 22, Line 1181, the authors explicitly admit that their proposed variance recovery fix (Eq. 17) "is a constructive alternative, not an additional component of the evaluated pipeline."

  2. Flawed Evaluation Protocol and Distorted Aggregate Statistics

  • 4-point ICP degeneracy: Each target curve is sampled at only n = 4 points (Page 5, Line 225; Page 28, Line 1506) and aligned via unconstrained rigid ICP before computing Chamfer distance, which is geometrically under-constrained and prone to degenerate local minima.
  • Unnormalized means completely dominated by a single class: In Table 2 (Page 32) and Table 6 (Pages 48-50), Parabola has a mean Chamfer error of ~853.3, whereas the other five shapes have errors between 1.1 and 8.6. Consequently, every unnormalized cross-shape average reported in the text (e.g., 140.00 vs. 148.50 on Page 6, Line 264, and Appendix T) is >98% dictated by Parabola alone.
  • Degradation on the strongest model: On Llama-3.3-70B (Table 1, Page 6), adding the paper's core Symbolic Regression module (FB + SR) actually degrades performance compared to standard text critique (FB) alone (0.634 -> 0.659 on 4-bar; 0.634 -> 0.648 on 6-bar).
Questions for authors
  1. To the Authors and Area Chair: How was a 50-page manuscript in this condition -- riddled with unresolved "??" and "(?)" macros, a completely missing "Table ??", half-written sentences ending in "is 6." and "on 7.", corrupted bibliography keys ("(son, 2024)", "(nob, 2022)"), duplicate Figure 5s, skipped Figures 17-20, and a core implementation y = f(x) that the paper's own appendix proves collapses to zero on closed curves -- submitted without basic human inspection, and why was a submission of this quality not Desk Rejected?
Reviewer_nQ2jOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents LLM-based agents for designing planar linkage mechanisms. Their method employs an LLM generating mechanism programs and then getting feedback through a simulator's output, distance from target, and symbolic regression, iteratively using that to refine the mechanism program. They contribute a novel benchmark, MSynth, and benchmark performance of their method relative to a classical linkage design algorithm. They analyze the effects of feedback on performance as well as the effects of symbolic regression feedback on reasoning in models.

Critical strengths

The paper presents, to my knowledge, novel work on LLM agents within this domain. The idea of using symbolic regression as an intermediary to assist with reasoning is compelling. While the area is somewhat narrow, it is a domain that has not received much attention, and so I think this is useful to consider in overall assessment.

The experimental and theoretical work here is substantial, creating a benchmark using it to evaluate several models with several key ablations. Re: overall assessment, it's important to note that there was considerable experimentation and theoretical work done.

Critical weaknesses

The experimental results provide weak evidence for the utility of the method. FB + SR loses to FB with the strongest model (Llama 70B) while beating FB with Qwen 4; both FB + SR and FB lose to baselines with Gemma 12B. A lot seems to depend on the model, and the overall method hurts with the strongest model. This is critical to my overall assessment: the paper represents primary contributions as a method including SR (indeed, that is what the title highlights) yet performance appears mixed.

The empiricism lacks any estimates of confidence or underlying model noise, so, on top of it not strongly supporting the method, it is hard to tell what to trust from these experiments. This is crucial for my overall assessment, as despite considerable experimentation, it is hard to know what to think of them.

Questions for authors

Could you provide a compute cost analysis of not simply token cost, but simulation, SR, etc., as well as baseline compute cost? Without these, it is difficult to say how meaningful the baseline comparison is, e.g., if the baseline is much cheaper per iteration, it is hard to say what the 120 vs 60 budget comparison means.

Could you say what you mean by grounded geometric verification/grounded analytical reasoning? An analysis of a shift towards this sort of grounded verification is described as a central contribution of this work, but any quantitative analysis is as far as I can tell only described in Appendix N.4, and it fails to define how you precisely did this analysis.

Reviewer_TT3ROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how numerical simulation results can be converted into feedback that helps language models improve mechanical designs. The system iteratively designs planar mechanisms whose output paths approximate target curves through generation, simulation, critique, revision, and memory retrieval. Its central method uses symbolic regression to fit analytical expressions to simulated trajectories, allowing the model to compare them with the target equations. Final design quality is still evaluated using the original simulated trajectories. Experiments on six curve families and three language models show relatively consistent benefits from the critique loop, while the additional benefits of symbolic feedback depend substantially on the model and task.

Critical strengths
  • The method is well motivated, with a clear connection between the feedback representation and the task structure. Compared with a single error score or a large collection of coordinates, converting simulated trajectories into expressions that can be compared with the target makes aspects of the geometry explicit and gives the model a more concrete basis for analysis. The implementation is straightforward and forms a reasonable design loop, using fitted expressions for guidance and the original simulations for scoring.

  • The experiments attempt to separate component contributions and report the method's limitations. The authors ablate critique, memory, and symbolic feedback, holding critique and memory settings fixed in key comparisons. They acknowledge that symbolic feedback does not benefit every model and report that GA outperforms all models when given a sufficiently larger budget. These results help clarify the method's role and limitations under restricted simulation budgets.

Critical weaknesses
  • The additional value of symbolic feedback over strong agents has not been sufficiently established. It is reasonable and expected that structured feedback incorporating task priors and facilitating comparisons can improve performance. The more important question is whether the proposed feedback bottleneck persists for strong agents capable of autonomously executing code, analyzing trajectories, extracting geometric features, or invoking fitting tools. The current experiments lack such baselines and therefore do not sufficiently establish whether symbolic feedback retains additional value for agents with more capable tool use.

  • Task coverage is limited, leaving the stability of the key gains unclear. The experiments include only six families of analytical curves, with five target instances per family. The benefits of symbolic feedback are also heterogeneous: in Table 1, Qwen improves more substantially, Gemma improves less, and Llama performs slightly worse when symbolic feedback is added. Although the authors acknowledge this model dependence, the limited task coverage is insufficient to establish the stability and scope of the gains. In addition, Table 1 does not provide confidence intervals for its aggregate metrics or the key differences, making the reliability of small improvements difficult to assess.

Questions for authors
  • Does symbolic feedback still produce consistent improvements with stronger models and agents capable of autonomously executing code, analyzing trajectories, and invoking fitting tools? Comparisons under the same simulation budget, together with the additional computational costs, would substantially strengthen the evidence for the method's added value.

  • Could the authors clarify the repeated-run and random-seed settings for each target instance and report confidence intervals for the aggregate metrics in Table 1 and the paired differences with and without symbolic feedback? Do these results support the stability of the observed improvements, rather than their being driven primarily by a small number of targets or stochastic generation differences?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.