- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HarnessSR optimizes reusable symbolic-regression agent harnesses while keeping language-model weights and the underlying SR algorithms fixed. It first optimizes role instructions and demonstrations, then searches workflow configurations and prompt assignments with brief adaptation, and finally refines instructions using SR-specific diagnostics of execution traces. The main configuration uses LLM roles to guide PySR. On 129 LSR-Synth tasks, the paper reports a mean ID Acc0.01 of 61.33% across three backbones, a 45.0% relative improvement over the next-best baseline, together with shorter per-task solving times. The appendices also evaluate OOD inputs, symbolic accuracy, and alternative SR backends.
Critical strengths
The method improves numerical accuracy across all three evaluated backbones. For Qwen3.6, ID Acc0.01 rises from 46.51% for SR-Scientist to 65.37%, while OOD Acc0.01 rises from 44.19% to 61.24% (Table 3 and Appendix F). These are meaningful gains, and the OOD results provide useful evidence that the benefit extends beyond the observed input range.
Critical weaknesses
-
Limited evidence of methodological novelty. The additional contribution over existing prompt and workflow optimization methods, such as MASS and GEPA, remains unclear. Direct comparisons under comparable budgets are needed.
-
Unmatched search budgets. HarnessSR uses 13 PySR calls per task, while the independent PySR baseline uses only one. Matched-budget comparisons and controlled ablations are needed to separate the benefit of optimized guidance from additional search effort.
-
Insufficient reproducibility and statistical details. The optimization score Q, diagnostic/reflection models and prompts, and repetition protocol need clearer specification. Variability across runs should also be reported.
-
Figure readability. Text in Figure 1 and several other figures and tables is too small, making them difficult to read. Please increase font sizes and improve legibility.
Questions for authors
See the weaknesses above.
4 comments
Author Response to Reviewer GnKD (2/3)
📝 W2 Comparison Under Matched Search Budgets (PySR & LLM generator)
We thank the reviewer for this constructive suggestion. As backend usage is optimized within the harness, we initially overlooked matching backend search budgets. We therefore added controlled experiments to isolate the effect of search effort.
1. PySR backend. Table R3.2 compares HarnessSR with two budget-matched controls: (1) PySR×13, which performs 13 independent searches of 125 iterations each and pools their candidates; and (2) PySR Long, which performs one continuous search of 1,625 iterations. Both match HarnessSR's total PySR iteration budget. Increasing PySR search effort yields only modest gains, and HarnessSR still maintains a substantial advantage under the same PySR budget.
2. LLM equation-generator backend. We conducted analogous comparisons with two budget-matched controls (Table R3.3). HarnessSR makes 13 backend calls, each with two rounds of eight generation requests. LLM Generator×13 performs 13 independent runs with the same per-run configuration and pools candidates, while LLM Generator Long performs 26 continuous rounds, each conditioned on the best equation found so far. HarnessSR outperforms LLM Generator×13 by 16.02 and 16.80 percentage points on ID and OOD, respectively.
Overall, Tables R3.2 and R3.3 demonstrate that HarnessSR's gains persist under matched backend search budgets, highlighting the benefits of agent-guided search coordination rather than additional search effort.
Table R3.2. Comparison under the same PySR search budget (LSR-Synth, 129 tasks).
| Method | PySR backend calls × iterations per call | ID [email protected] (%) ↑ | OOD [email protected] (%) ↑ | SA (%) ↑ |
|---|---|---|---|---|
| PySR | 1*125 | 32.82 | 31.78 | 5.17 |
| PySR×13 (independent restarts with candidate pooling) | 13 × 125 iterations | 40.83 | 37.21 | 7.24 |
| PySR Long (continuous search) | 1 × 1,625 iterations | 37.98 | 34.63 | 6.72 |
| HarnessSR / Qwen3.6-35B-A3B | 13 × 125 iterations | 65.37 | 61.24 | 10.34 |
| HarnessSR / Qwen3-Coder-30B-A3B-Instruct | 13 × 125 iterations | 57.36 | 55.04 | 12.66 |
| HarnessSR / GPT-OSS-20B | 13 × 125 iterations | 61.24 | 59.69 | 11.63 |
Table R3.3. Comparison under the same LLM equation-generator search budget (LSR-Synth, 129 tasks). LLM backbone: Qwen3.6-35B-A3B.
| Method | LLM backend calls × rounds per call | ID [email protected] (%) | OOD [email protected] (%) | SA (%) ↑ |
|---|---|---|---|---|
| LLM Generator×13 | 13 × 2 | 43.41 | 41.34 | 7.75 |
| LLM Generator Long | 1 × 26 | 42.12 | 37.47 | 5.43 |
| HarnessSR | 13 × 2 | 59.43 | 58.14 | 13.44 |
Author Response to Reviewer GnKD (3/3)
📝 W3 Clarification on Reproducibility and Statistical Details
We thank the reviewer for this valuable suggestion and apologize for the lack of clarity. We clarify the optimization objective, the diagnosis and reflection mechanisms, and the repetition protocol below, and provide additional statistical results. We will explain these details more clearly in the revised manuscript.
1. Optimization objective . For each development task , we use the negative average of the final equation's normalized mean squared error (NMSE) on the held-out ID and OOD data as the quality score:
All three optimization stages use this quality objective to evaluate and optimize the harness.
2. Diagnosis and reflection. We apologize for omitting further details about the reflection model from the submitted manuscript. In HarnessSR, diagnosis and reflection use the same LLM as the solver backbone, rather than a different model. The diagnostic prompt instructs the model to perform four SR-specific checks on failed execution traces, identify problems, provide supporting evidence, and specify the affected roles. Based on this diagnostic evidence, the reflection prompt asks the model to analyze and revise role instructions through four steps: Observed, Supported By, Instruction at Fault, and Change. The diagnostic and reflection prompts have been added to Appendices G.3 and G.4 of the revised manuscript, respectively.
3. Repetition protocol. We apologize for not clarifying that the main experiments in the original submission were conducted with three random seeds. We have added the mean ± standard deviation (SD) of the main results in Appendix J (Table 17) of the revised manuscript.
4. Additional experiments on performance variability. We added experiments to further assess variability in harness optimization and inference.
(1) Variability in harness optimization. For each LLM backbone, we ran the full harness optimization procedure independently three times on the same training data. Each resulting harness was then evaluated on the same test set with three random seeds. Tables R3.4 and R3.5 report the mean ± SD. Comparing the three final harnesses, we found that two used workflow (1, 3, 3, 0, 1), with three Diagnoser–Repairer rounds and no cross-branch discussion. The third, which performed slightly worse, used workflow (1, 3, 2, 1, 1), with two Diagnoser–Repairer rounds and cross-branch discussion enabled. Although the resulting harnesses are not identical, their final performance differs only modestly, indicating that optimization randomness had a limited effect on performance in these runs.
(2) Variability in inference with a fixed harness. Tables R3.4 and R3.5 report the mean ± SD over three seeds for each final harness, showing relatively stable performance when the harness is held fixed. The main experimental results in the updated manuscript also report mean ± SD over three seeds.
Table R3.4. Performance of three independently optimized harnesses, each evaluated with three random seeds. LLM backbone: Qwen3.6-35B-A3B.
| Harness | ID [email protected] (%) ↑ | OOD [email protected] (%) ↑ | SA (%) ↑ |
|---|---|---|---|
| 1 | 65.37 ± 1.95 | 61.24 ± 2.05 | 10.34 ± 0.90 |
| 2 | 62.27 ± 2.24 | 60.47 ± 1.55 | 9.82 ± 1.18 |
| 3 | 64.34 ± 1.34 | 60.21 ± 2.49 | 11.11 ± 0.45 |
| Across three optimization runs | 63.99 ± 1.58 | 60.64 ± 0.54 | 10.42 ± 0.65 |
Table R3.5. Performance of three independently optimized harnesses, each evaluated with three random seeds. LLM backbone: GPT-OSS-20B.
| Harness | ID [email protected] (%) ↑ | OOD [email protected] (%) ↑ | SA (%) ↑ |
|---|---|---|---|
| 1 | 61.24 ± 1.55 | 59.69 ± 2.33 | 11.63 ± 0.78 |
| 2 | 58.91 ± 0.78 | 57.11 ± 1.18 | 10.59 ± 0.45 |
| 3 | 62.02 ± 3.38 | 58.66 ± 2.72 | 11.89 ± 0.45 |
| Across three optimization runs | 60.72 ± 1.62 | 58.49 ± 1.30 | 11.37 ± 0.69 |
📝 W4 Figure and Table Readability
We apologize for the difficulty in reading the figures and tables. We will increase the font sizes in the revised manuscript to improve overall readability.
Dear Reviewer,
Thank you again for your thoughtful review. We have provided detailed responses to your comments and added experiments to address your concerns. As the discussion deadline is approaching, we would greatly appreciate any further feedback you may have, particularly on whether our responses and new results address your main concerns. We would be happy to clarify any remaining questions.
Thank you for your time and consideration.
Author Response to Reviewer GnKD (1/3)
We thank the reviewer for the constructive feedback and for recognizing the consistent accuracy gains across backbones and on OOD inputs. Below, we address your concerns about methodological contributions, budget-matched comparisons, reproducibility, and figure readability point by point.
📝 W1 Methodological Novelty and Budget-Matched Comparisons
We thank the reviewer for this valuable feedback. We explain HarnessSR's four methodological contributions from two perspectives and present additional experiments to evaluate their effectiveness.
1. Contributions to general harness optimization. MASS, GEPA, and HarnessSR all optimize the harness outside the LLM, but use different mechanisms. MASS optimizes prompts and workflows separately, while GEPA uses reflection to optimize prompts only. We consider prompts and workflows strongly coupled, so optimizing either separately does not guarantee an optimal combination. Therefore, HarnessSR proposes:
(Innovation 1) Joint optimization of role prompts and workflows. We define a joint search space for role prompts and common workflows and perform an initial joint optimization of both.
(Innovation 2) Brief prompt adaptation further reduces the mismatch between locally optimized prompts and newly selected workflows.
2. Contributions specific to SR. MASS and GEPA were not designed for SR and do not include SR-specific mechanisms. Beyond defining roles and workflow topologies suited to SR, HarnessSR introduces two further designs:
(Innovation 3) Bidirectional adaptation between SR search guidance and backend feedback. Dedicated interfaces for PySR and an LLM-based equation generator translate structural hypotheses and search plans into executable backend guidance, and feed candidate equations, residuals, and search results back into harness optimization.
(Innovation 4) Instruction optimization driven by SR diagnostics. We build on and adapt GEPA's trajectory reflection mechanism by adding structured SR diagnostics and targeted role optimization. These diagnostics examine evidence-to-action alignment, structural exploration, constraint justification, and expression quality, identify the roles needing improvement, and guide revisions to their instructions. This turns solving experience into search guidance that can be reused across tasks.
Compare HarnessSR with MASS-SR and GEPA-SR. To directly evaluate the combined effect of these designs, we added the comparisons in Table R3.1. We adapted MASS and GEPA to SR using the same backend interfaces introduced in Innovation 3, and all three methods use Qwen3.6-35B-A3B as the backbone. Because the methods differ substantially, it is difficult to match exactly the number of solver executions used for optimization feedback. We therefore compare them under optimization budgets that are as close as possible. Both MASS-SR and GEPA-SR use the SR role set defined in our paper. Since GEPA-SR optimizes prompts only, it uses HarnessSR's final workflow shown in Figure 7 of the submitted manuscript.
Under comparable budgets of complete solver executions, HarnessSR exceeds MASS-SR and GEPA-SR in ID [email protected] by 16.53 and 12.92 percentage points, in OOD [email protected] by 14.47 and 13.18 percentage points, and in SA by 2.85 and 4.14 percentage points, respectively. HarnessSR also uses fewer model requests, input tokens, and output tokens than MASS-SR. Compared with GEPA-SR, it uses more model requests and input tokens but fewer output tokens. Overall, HarnessSR achieves better equation recovery under comparable solver-execution budgets, demonstrating the effectiveness of its optimization designs for SR.
We will include these comparisons in the revised manuscript and clarify HarnessSR's methodological contributions.
Table R3.1. Performance comparison of HarnessSR, MASS-SR, and GEPA-SR. MASS-SR and GEPA-SR are partial adaptations of MASS and GEPA to SR.