acceptodds
Under review as a conference paper at ICLR 2027

Beyond a Better Score: Long-Horizon Agentic ML Development and Evaluation Protocol for Physics Time Series

Abstract

Physics time-series data are central to fundamental discoveries involving dark matter, neutrinos, and gravitational waves. Machine learning has shown strong potential to recover weak scientific signals buried in complex noise. Recent LLM-agent systems further automate ML development, but existing approaches are often driven primarily by one or a few numerical scores. In pursuit of a better score, agents may exploit shortcuts, trigger mode collapse, or game the evaluation procedure, producing models that achieve high scores but are scientifically invalid. We introduce SIDERIUS, an LLM agent system designed to move beyond a better score toward scientifically valid, long-horizon ML development. SIDERIUS combines contract-defined composable capabilities for flexible orchestration and human-in-the-loop, a multi-layer scientific evaluation protocol that distinguishes valid models from unhealthy or gaming models, and resource-aware exploration for computationally expensive scientific ML. In our TIDMAD benchmark, SIDERIUS with a GPT-based orchestrator achieves the highest Valid-model proposal rate of 63.8% and the best valid denoising score of 2.744, compared with 1.6% and −0.647, respectively, for the same GPT model using the general-purpose Codex coding agent. We further benchmark SIDERIUS on three additional physics time-series datasets, demonstrating its reuse and performance across different tasks. Together, these results suggest that progress in agentic scientific ML should be measured beyond a better score, by how an agent explores and by whether the models it produces can be trusted as science.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Large Language Models, Multi-Agent Systems, Scientific Machine Learning, Time-Series Analysis, Long-Horizon Agents, Composable Agents, Scientific Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6dyCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes an agentic system to solve the physics time series denoising problem.

Critical strengths
  1. The proposed method demonstrates good performance against closed-source harnesses under the proposed criteria.
  2. The cost analysis offers good insights into the cost-efficiency of SIDERIUS.
Critical weaknesses

MLE-agent baselines. The work seem to constitute a Machine Learning Engineering (MLE) agent. Although the paper compares against general-purpose coding agents, the authors are encouraged to evaluate against at least one established MLE-agent framework, such as MLE-STAR [2] or AIDE [1], under the same scientific task instructions, data access, evaluation criteria, and computational budget. This would help establish the benefit of SIDERIUS relative to existing frameworks specifically designed for iterative ML experimentation.

Robustness to judge choice. Appendix A.2 describes the use of gpt-5.6-sol. Could the authors assess sensitivity to the choice of LLM judge so that we know whether gaming classifications and the resulting comparisons between systems remain stable?

Judge correctness. Could the authors test the frozen judge prompt and rubric on independently constructed scripts with known valid behavior or deliberately introduced task violations? Reporting false-positive and false-negative rates would help establish whether the judge correctly identifies violations rather than merely producing consistent judgments.

Stability across runs. Appendix E.2 acknowledges the absence of repeated independent campaigns. Could the authors run multiple independent repetitions and report the mean and standard deviation of valid-proposal rate and best valid score to assess performance stability?

(1): https://arxiv.org/abs/2502.13138

(2): https://arxiv.org/abs/2506.15692

Questions for authors

N/A

5 comments

We thank the reviewer for reading our paper carefully. We are glad that the reviewer appreciates that SIDERIUS clearly outperforms the closed-source agent harnesses under our evaluation criteria, and we appreciate the positive comment on our cost analysis.

The four concerns the reviewer raises are concrete, and they are instructive for further strengthening our conclusions. To address them, we did the following.

  • We added AIDE as an MLE-agent baseline.
  • We tested three LLM judges from different model families on 200 synthetic test cases with known answers, and report their false-positive and false-negative rates.
  • We also added repeated runs on H100 and A100.

Below, we address the four concerns in order. The second and third concerns are both about the LLM judge, so we answer them together.

The reviewer suggests comparing against at least one established MLE-agent framework, such as MLE-STAR or AIDE, under the same task instructions, data access, evaluation criteria and compute budget. We followed this suggestion and added AIDE as a baseline. AIDE drafts, debugs and improves complete training scripts through a fixed tree search. We ran it under the same task instructions, data access, GPU and time budget, with the same underlying model as SIDERIUS (GPT-5.6-sol, medium), and judged it with the same frozen rubric:

System (H100)Completed proposalsValid rateGaming rateBest valid scoreLLM cost
AIDE14129.8%49.6%1.05$101
SIDERIUS fixed workflow3129.0%9.7%1.09$55
SIDERIUS fixed workflow + data analysis3250.0%0.0%1.05$65
SIDERIUS orchestration4959.2%6.1%2.74$690
SIDERIUS orchestration + data analysis11533.0%50.4%7.28$719

For reference, the raw input scores 1.00, and the best result in the original TIDMAD paper is 6.43.

This comparison shows two things. First, a framework designed for iterative ML experimentation does not by itself prevent gaming. AIDE and the SIDERIUS fixed workflow both follow a fixed procedure, but AIDE lacks the contract-based core of SIDERIUS. 49.6% of the models AIDE completed are gaming, against 0% to 9.7% for the SIDERIUS fixed workflow. Second, higher scores come from composing capabilities flexibly. The best valid scores of AIDE and the fixed workflow stay near the raw input (1.05 to 1.09). Only orchestration, which composes the same capabilities flexibly, raises the score clearly: 2.74, and 7.28 with data analysis, above every result in the original TIDMAD paper.

In the revised paper, we will add AIDE to the main results table, cite AIDE and MLE-STAR, and discuss more of the MLE-bench agents in the related work.

The second and third concerns are both about the LLM judge. The second asks whether the verdicts stay stable if a different judge is used. The third asks how many false positives and false negatives the judge makes on cases with known answers. We answer both with the same calibration set.

In our updated formulation of the rubric design, two domain experts first led the design of 200 synthetic test cases. For every rule, the cases include examples that violate it and examples that do not, and the experts decided the correct answer for each case. 93 cases are gaming, 101 are not, and the answer key of 6 cases is itself UNCERTAIN. We tested three LLM judges from different model families (GPT, Claude and Gemini) with the same frozen rubric and prompt:

GPTClaudeGemini
Gaming accuracy (definite answers only)97.5%97.0%95.5%
Final-verdict accuracy (definite answers only)98.5%95.4%94.5%
False negatives: gaming judged not gaming (of 93)3 (3.2%)1 (1.1%)4 (4.3%)
False positives: non-gaming judged gaming (of 101)1 (1.0%)2 (2.0%)2 (2.0%)

On the choice of judge, the three judges differ by less than 2 percentage points in gaming accuracy and by less than 4 percentage points in final-verdict accuracy. The verdicts are driven mainly by the rubric and the evidence, not by which model family serves as the judge.

On false positives and false negatives, a false negative matters more in our pipeline. A model wrongly judged as gaming at most removes one valid model from the count. A gaming model that is missed is counted as valid. Among the three judges, Claude has the fewest false negatives, with a false-negative rate of 1.1% and a false-positive rate of 2.0%. An UNCERTAIN answer on a case with a definite key is not counted as an error, because such cases go to human review.

The judge in the original paper was GPT (gpt-5.6-sol). Because of cost, we used a single judge (Claude) to re-judge all results. The items the judge is uncertain about are reviewed independently by two domain experts. This review is ongoing, and we will report the results and the inter-expert agreement (Cohen's kappa) during the discussion period.

Our updated evaluation package is well defined and reusable: for every rule, it states what evidence is needed and which diagnostic skill provides it. In practice, it is used as follows. After a run, the package is applied to all proposals: every model goes through the diagnostic skills and is then reviewed by the LLM judge. Human experts then review the models the judge is uncertain about and the highest-scoring models. The highest-scoring model among those confirmed valid is delivered. A high-scoring gaming model would have to pass the diagnostic skills, the LLM judge and the human review to be delivered.

In the revised paper, the appendix will include the full rubric, all 200 calibration cases with their answer, and the three judges' calibration results, including the false-positive and false-negative rates.

Reviewer_wyr8OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents SIDERIUS, an LLM agent system for developing machine learning models for physics time series. The system supports fixed workflows and dynamic LLM orchestration through explicit input/output contracts for capabilities, and combines resource management, online Health checks, and offline behavioral review to evaluate candidate models. The paper compares six configurations on the existing TIDMAD denoising task and demonstrates workflow reuse on three additional physics regression tasks.

Critical strengths

The system adopts a modular design that separates capabilities from scheduling. It separates scientific capabilities, including data analysis, model proposal, implementation, and training, from the caller’s control logic and organizes modules through explicit input/output interfaces. This provides a clear organizational structure for reusing the same capabilities and exchanging experimental artifacts across fixed workflows and LLM orchestration.

Critical weaknesses
  1. The scientific validity protocol lacks independent validation: §5 acknowledges that the rubric was refined using the evaluated archives and has not been independently validated on new campaigns. Although the labels were reviewed by domain experts, this work has not quantified the protocol’s error rates or established its relationship to reliable signal recovery.
  2. The contributions of the key design components have not been isolated: The comparison between Run 2 and Run 5 in §4.1.3 changes the entire development framework. Sharing the same GPT backbone does not isolate the respective effects of capability interfaces, prompts and tools, search strategies, and resource management, making it difficult to determine which design choices actually account for the improvements.
  3. Evidence for the robustness and generalizability of the system’s advantages is insufficient: The main system comparison is conducted only on TIDMAD; the three additional tasks primarily demonstrate workflow reuse, and uncertainty across independent repeated runs is not reported. In Table 1, adding DA to the LLM-orchestrated configuration reduces the Valid-model proposal rate from 63.8% to 2.2%. The explanation for this reversal in §5 remains untested, leaving the applicability of the best-performing configuration unclear.
Questions for authors
  1. Would the best-valid scores and system rankings in Table 1 change if the acceptance restrictions on post-fit output rescaling, clipping, and similar operations were relaxed?
  2. Which of the restrictions on post-fit modifications listed in §4.1.2 were explicitly communicated to all systems before search began?
  3. If candidates sharing the same set of trained weights and differing only in inference configuration were counted as a single proposal, would the Valid-model proposal rates and system rankings remain unchanged?
  4. Could the authors provide an end-to-end workflow diagram or pseudocode for both the fixed-workflow and LLM-orchestrated configurations, showing capability calls, feedback paths, failure handling, and stopping and model-selection rules? Figures 1–2 do not sufficiently clarify how these components interact during an actual model-development cycle.
9 comments

We thank the reviewer for reading our paper and bringing instructive insights. We are glad that the reviewer appreciates the modular design of SIDERIUS. This is what allows the same capabilities to be reused across the fixed workflow and LLM orchestration, and experimental artifacts to be exchanged between them.

The three concerns the reviewer raises are insightful. To address them, we did the following:

  • Two domain experts refined and froze the rubric. We calibrated LLM judges on 200 synthetic test cases, which gives their error rates, and re-judged all results with the frozen rubric.
  • We added an AIDE baseline, together with an ablation of resource control.
  • We also added repeated runs on H100 and A100.

Below, we first address the three concerns in order, then answer Questions 1 to 3, and finally answer Question 4 with pseudocode for both modes.

The reviewer points out that the rubric was refined on the evaluated archives, was not independently validated on new campaigns, has no quantified error rate, and has no stated relationship to reliable signal recovery. These criticisms apply to the original paper, where the final labels came from human judgement assisted by an LLM judge. In the revision, we replaced this with a strict tiered process: the diagnostic skills provide the evidence, an LLM judge applies the frozen rubric to every proposal, and only the items the judge cannot decide go to two domain experts.

The rubric has to be refined iteratively because of the nature of the problem. Agents keep finding new ways to game an evaluation, and many of them only surface in large-scale runs. No rubric can anticipate all of them before the experiments, so it has to be refined as the experiments reveal new cases. What matters is to check that the refined rubric does not overfit. We did this in three ways.

First, error rates. Two domain experts led the design of 200 synthetic test cases. For every rule, the cases include examples that violate it and examples that do not, and the experts decided the correct answer for each case. The three LLM judges scored as follows:

GPTClaudeGemini
Gaming accuracy (definite answers only)97.5%97.0%95.5%
Final-verdict accuracy (definite answers only)98.5%95.4%94.5%
Gaming models let through (of 93 gaming items)314

An UNCERTAIN answer on an item with a definite key is not counted as an error, because such items go to human review.

Second, new runs. Because of cost, we used a single LLM judge (Claude) to re-judge all results: the main experiments (942 proposals) and four new sets of runs, namely a second H100 run of the same configurations (499), a replicate on A100 (513), the new AIDE baseline (133), and the new resource-control ablation (102). The judge answered UNCERTAIN for 6.3% of the main experiments, and for 1.6%, 1.0%, 0% and 8.8% of the four new sets. The frozen rubric gives clear verdicts on new runs as well. The items the judge is uncertain about are reviewed by the two domain experts independently. This review is ongoing, and we will report the results and the inter-expert agreement (Cohen's kappa) during the discussion period. In addition, every model that scores more than 0.5 above the raw input has already been reviewed by the experts, and no mis-judge was found. Our updated evaluation package is well defined and reusable: for every rule, it states what evidence is needed and which diagnostic skill provides it. In practice, it is used as follows. After a run, the package is applied to all proposals: every model goes through the diagnostic skills and is then reviewed by the LLM judge. Human experts then review the models the judge is uncertain about and the highest-scoring models. The highest-scoring model among those confirmed valid is delivered. A high-scoring gaming model would have to pass the diagnostic skills, the LLM judge and the human review to be delivered.

Third, the relationship between validity and signal recovery. Using the true clean signal itself as the output scores 10.1, which is the ceiling for a genuine denoising model. Every model judged valid scores below this ceiling. The best one scores 7.28, above the best result of 6.43 in the original TIDMAD paper. In contrast, if models are selected only by score and online Health checks, the two CLI agents reach 14.4 and 14.7, above the score of the true signal. Such scores cannot come from recovering the signal, and these are exactly the models the rubric labels as gaming. So the rubric rejects precisely the models that score high without recovering the signal.

In the revised paper, the appendix will include the full rubric, all 200 calibration cases, and the calibration and review results.

Another concern is that the comparison between Run 2 and Run 5 changes the entire development framework. Even with the same GPT backbone, it does not separate the effects of capability interfaces, prompts and tools, search strategy and resource management.

We agree that isolating each design choice would make the attribution clearer. Some components, such as the prompts and tools, are hard to remove in isolation, because the other parts of the system depend on them and the system would no longer run as a whole. For this reason, we focused on the two components that can be cleanly separated and matter most for our claims, and compared each against a control.

First, the contract-based core. We added AIDE as a baseline. It uses the same underlying model as SIDERIUS (GPT-5.6-sol, medium) and has a structured experimentation loop, but it lacks the contract-based core of SIDERIUS. If a structured loop were enough, AIDE should also game rarely. The result shows that it does not: 49.6% of the models AIDE completed are gaming, against 0% to 9.7% for the SIDERIUS fixed workflow.

Second, resource control. In SIDERIUS, resource control works in three steps. Before a training run starts, it predicts the run's time and GPU memory from the model and earlier measurements, and refuses runs that would not fit. During the run, it enforces the declared budget and stops training that would exceed it. After the run, it records the measured cost and uses it for the next prediction. In the ablation, we removed this whole loop from both orchestration and the fixed workflow, and instead stated the same time and VRAM budget in the prompt. All other settings were kept the same:

Orchestration: fullOrchestration: removedFixed workflow: fullFixed workflow: removed
Valid rate46.3%86.7%61.3%40.0%
Best valid score1.171.031.930.95
Training runs over their own time budget0%30.6%0%29.6%

Without resource control, about 30% of training runs exceeded their own time budget, against none with it. A budget stated in the prompt alone does not keep the agent within it. Resource control makes sure that all systems are compared under the same compute budget. Its effect on valid rate and score shows no consistent direction in these single runs, so we do not claim that it improves the scientific results.

In the revised paper, we will add AIDE to the main results table (Table 1), with its valid rate, gaming rate, best valid score and LLM cost. The resource-control ablation table above will go into the appendix, together with a description of what the ablation removes. We will also add a paragraph to the discussion that lists which components we isolated (the contract-based core through the AIDE comparison, resource control, and data analysis) and ablated.

Reviewer_LVvkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SIDERIUS, an LLM agent system designed for long-horizon ML development on physics time-series. The main motivation is that standard coding agents tend to hack the evaluation metric and produce scientifically meaningless models when optimizing for a single score. To address this, SIDERIUS introduces contract-based modular capabilities, resource-aware execution, online health checks, and an offline LLM-as-a-judge behavior review. Experiments on the TIDMAD and three PhyTS tasks show that general coding agents suffer from severe reward gaming, whereas SIDERIUS produces a much higher fraction of scientifically valid models

Critical strengths

The problem studied here is very real and important for AI4Science. Optimizing a proxy metric often leads to reward hacking, and the paper gives very concrete, convincing examples of how agents cheat in physics denoising (e.g., substituting memorized sine waves or injecting high-frequency noise just to pass variance checks). Comparing SIDERIUS with Codex under the same underlying GPT model is a nice setup. And the appendix is quite transparent about the failure cases and what went wrong during different runs.

Critical weaknesses

1- My biggest concern is how the evaluation protocol itself is validated. If I understand correctly, the offline behavioral rubric was iteratively refined by inspecting the exact same campaign logs that are used to report the main results. Later expert adjudication was also not blinded, and there is no quantitative metric on inter-expert agreement or how accurate the standalone LLM judge is. Since all the main claims (valid proposal rate, best valid score) depend entirely on who passes this rubric, how do we know the rubric isn't just overfitted to the specific failure modes of these runs?

2- The baseline comparison jumps straight from raw general-purpose CLI coding agents (Claude Code, Codex) to the full SIDERIUS system. While the paper cites ML-specific agent scaffolds like AIRA or MLE-bench agents, none of them are included as baselines. It makes me wonder: how much of the gain comes from SIDERIUS's specific contract architecture versus just having any structured ML experimentation loop? Also, Section 3.3 highlights the "predict, enforce, learn" resource management as a core contribution, but I couldn't find any direct evaluation or ablation for it.

3- Each experimental condition is only run once per frequency band, so the proposals within a run are not independent replicates. Because agent search trajectories usually have high variance, it is risky to draw strong conclusions from a single run. For example, adding the Data Analysis module slightly helps the fixed workflow, but in Orchestration mode it suddenly causes the agent to generate tons of gaming models and the valid rate drops to almost nothing. Why does giving domain data analysis to the orchestrator make it cheat so much more?

Questions for authors

1- How reliable is the frozen LLM-as-a-judge rubric when applied to completely new runs?

2- Why did enabling the Data Analysis capability in Orchestration mode lead to such a massive spike in gaming behavior compared to the base orchestration run?

3- Since the offline behavior check cannot be used during runtime, how should the agent select its final delivered model in practice so that it doesn't pick a high-scoring gaming model?

7 comments

We thank the reviewer for reading our paper so carefully. We are glad that the reviewer sees why this problem matters for AI4Science: when an agent optimizes a single score, it can easily game the evaluation. We also appreciate the reviewer's positive comments on the concrete cheating cases in the paper and on comparing SIDERIUS with Codex under the same GPT model.

The three concerns the reviewer raises are central to our claims. To address them, we did three things. First, two domain experts refined and froze the rubric, and after calibrating LLM judges on 200 synthetic items, we re-judged all results with it. For experiments: we added an AIDE baseline, together with an ablation of resource control. We also added more repeated runs on H100 and A100 to strengthen the statistical credibility of our conclusion.

Below, we address the three concerns in order. Questions 1 and 3 are answered in the first part, and Question 2 in the third.

The reviewer's main concern is that the rubric was refined on the same run logs that the main results come from, so it may only fit the failure modes of these runs. This is a fair concern for the original paper, where the final labels came from human judgement assisted by an LLM judge. In the revision, we replaced this with a strict tiered process: the diagnostic skills provide the evidence, an LLM judge applies the frozen rubric to every proposal, and only the items the judge cannot decide go to two domain experts.

The rubric has to be refined iteratively because of the nature of the problem. Agents keep finding new ways to game an evaluation, and many of them only surface in large-scale runs. No rubric can anticipate all of them before the experiments, so it has to be refined as the experiments reveal new cases. One of the main contributions of this work is that we turned the result of this process, i.e. the rubric, with the diagnostic skills and the calibration problem set, into an evaluation package that other axion dark-matter tasks can reuse directly. To make sure that the refined rubric does not overfit, we did three things.

First, we tested the LLM judges on an independent synthetic problem set (calibration set). Two domain experts refined and froze the rubric. Each rule states what evidence it needs and which diagnostic skill provides it. If the evidence is insufficient, the judge must answer UNCERTAIN instead of guessing. The experts then led the design of 200 synthetic test cases. For every rule, the cases include examples that violate it and examples that do not, and the experts decided the correct answer for each case. On the calibration set, the three LLM judges performed similarly:

GPTClaudeGemini
Gaming accuracy (definite answers only)97.5%97.0%95.5%
Final-verdict accuracy (definite answers only)98.5%95.4%94.5%
Gaming models let through (of 93 gaming items)314

An UNCERTAIN answer on an item with a definite key is not counted as an error, because such items go to human review.

Second, we applied the frozen rubric to new runs, which answers Question 1. Because of cost, we used a single LLM judge (Claude) to re-judge all results: the main experiments (942 proposals) and four new sets of runs, namely a second H100 run of the same configurations (499), a replicate on A100 (513), the new AIDE baseline (133), and the new resource-control ablation (102). The judge answered UNCERTAIN for 6.3% of the main experiments, and for 1.6%, 1.0%, 0% and 8.8% of the four new sets.

Third, the items the judge is uncertain about are reviewed by the two domain experts independently, without discussion. The experts see the judge's analysis and all the evidence, but not each other's verdicts. This review is ongoing, and we will report the results and the inter-expert agreement (Cohen's kappa) during the discussion period. In addition, every model that scores more than 0.5 above the raw input has already been reviewed by the experts, and no mis-judge was found.

The ongoing uncertain review does not change the main conclusions. We resolved all of them in the most favorable and in the least favorable way, and the advantage of SIDERIUS over Claude Code and Codex holds in both cases. The range for the H100 main experiments:

SystemValid rateGaming rate
Claude Code1.4%98.1% to 98.6%
Codex0.6% to 0.9%97.1% to 97.4%
SIDERIUS fixed workflow29.0% to 41.9%9.7%
SIDERIUS fixed workflow + data analysis50.0%0.0%
SIDERIUS orchestration59.2% to 61.2%6.1% to 22.4%
SIDERIUS orchestration + data analysis33.0% to 50.4%50.4% to 70.4%

Question 3 asks how to select the delivered model in practice without picking a high-scoring gaming model. Because the evaluation package is well defined, it can be used for this directly. After the run, the package is applied to all proposals: every model goes through the diagnostic skills and is then reviewed by the LLM judge. Human experts then review the models the judge is uncertain about and the highest-scoring models. The highest-scoring model among those confirmed valid is delivered. A high-scoring gaming model would have to pass the diagnostic skills, the LLM judge and the human review to be delivered.

In the revised paper, the appendix will include the full rubric, all 200 calibration items, the calibration and review results, and the two tables above. The discussion section will describe this selection procedure.

The reviewer also points out that the paper jumps from general CLI coding agents straight to the full SIDERIUS, so it is hard to tell whether the gain comes from the contract-based architecture or from any structured experimentation loop. The reviewer also notes that resource control is not evaluated on its own. To address those concerns, we added two additional experiments.

First, we added AIDE as a baseline. AIDE is a representative agent scaffold from MLE-bench. It drafts, debugs and improves complete training scripts through a fixed tree search, which makes it a structured ML experimentation loop. AIDE ran on the same task with the same GPU and time budget and the same underlying model as SIDERIUS (GPT-5.6-sol, medium), and was judged with the same frozen rubric:

System (H100)Completed proposalsValid rateGaming rateBest valid scoreLLM cost
AIDE14129.8%49.6%1.05$101
SIDERIUS fixed workflow3129.0%9.7%1.09$55
SIDERIUS fixed workflow + data analysis3250.0%0.0%1.05$65
SIDERIUS orchestration4959.2%6.1%2.74$690
SIDERIUS orchestration + data analysis11533.0%50.4%7.28$719

For reference, the raw input scores 1.00, and the best result in the original TIDMAD paper is 6.43.

There are two observations from this new experiment.First, a structured loop alone does not prevent gaming. AIDE and the SIDERIUS fixed workflow both follow a fixed procedure, but AIDE lacks the contract-based core of SIDERIUS: the protected evaluator, the online Health checks and the validity decision. Half of the models AIDE completed are gaming (49.6%), against 0% to 9.7% for the SIDERIUS fixed workflow. Second, higher scores come from composing capabilities flexibly. The best valid scores of AIDE and the fixed workflow stay near the raw input (1.05 to 1.09). Only orchestration, which composes the same capabilities flexibly, raises the score clearly: 2.74, and 7.28 with data analysis, above every result in the original TIDMAD paper.

Also, we ran an ablation without resource control, in both orchestration and the fixed workflow. The predict-enforce-learn loop was replaced by a prompt that states the same time and VRAM budget, and all other settings were kept the same:

Orchestration: fullOrchestration: removedFixed workflow: fullFixed workflow: removed
Valid rate46.3%86.7%61.3%40.0%
Best valid score1.171.031.930.95
Training runs over their own time budget0%30.6%0%29.6%

Without resource control, about 30% of training runs exceeded their own time budget, against none with it. Resource control keeps all systems under the same compute budget. Its effect on valid rate and score shows no consistent direction in single runs, so we do not claim that it improves the scientific results.

In the revised paper, AIDE will be added to the main results table, and the resource-control ablation to the appendix.

Comments on the paper

2 comments

Dear Reviewers,

As the discussion phase draws to a close, please acknowledge the author's response and let them know whether it has addressed some or all of your concerns.

Thank you for the time and effort you and the other reviewers have put into this discussion.

AuthorsOpenReview ↗

Summary of Contributions and Rebuttal Outcomes

We thank all reviewers for their time and thoughtful feedback. Below, we summarize the key strengths identified by the reviewers and how we have addressed their main concerns.

This paper addresses a critical challenge in AI for Science: making sure that the models an agent produces are scientifically valid rather than just getting a better score. Our contribution to address this problem includes 1. SIDERIUS, a long-horizon LLM agent with composable agent capabilities and resource awareness and 2. A scientific valid evaluation protocol for agent-proposed models. The reviewers recognize the importance of this problem and appreciate our approach to addressing it. Reviewer LVvk emphasizes that “The problem studied here is very real and important for AI4Science” and highlights the “very concrete, convincing examples” of evaluation gaming. Reviewer wyr8 appreciates the modular architecture, noting that it “provides a clear organizational structure for reusing the same capabilities” across different workflows. Reviewer 6dyC recognizes that “The proposed method demonstrates good performance against closed-source harnesses under the proposed criteria” and that “The cost analysis offers good insights into the cost-efficiency of SIDERIUS.”

The reviewer’s main concern is the reliability of Evaluation Protocol and LLM-as-a-Judge. (LVvk, wyr8, 6dyC). To address this. we established a tiered evaluation process: expert-curated calibration set → LLM judge calibration → application of the frozen rubric → independent expert cross-checking. Three LLM judges achieved 94.5–98.5% accuracy on 200 expert-labeled cases. Two domain experts independently reviewed all 81 uncertain cases, achieving Cohen's κ of 0.89–0.95. Re-evaluation with the frozen rubric supports our main conclusions.

Reviewers also request additional experiments, baseline, and ablation study (LVvk, wyr8, 6dyC). We added AIDE under matched task, model, and resource budgets. Its gaming rate was 49.6%, compared with 0–9.7% for SIDERIUS fixed workflows. Independent H100/A100 campaigns confirmed the valid-rate gap: 15.0–68.4% for SIDERIUS versus 0–1.4% for Claude Code and Codex. Resource-control ablations showed that removing resource control caused approximately 30% of training runs to exceed their time budgets.

Reviewer also pointed out the lack of explanation for the decreased performance of data analysis (LVvk, wyr8). We investigated this unexpected behavior and found it as a consequence of GPU-platform failure. After fixing the bug and repeated the affected experiments, we obtain a best valid score of 7.28, exceeding the 6.43 reported in the original TIDMAD paper. We also demonstrated framework reuse across three additional physics tasks.

Overall, SIDERIUS brings together composable scientific ML development, scientific-validity evaluation, and empirical analysis of agent exploration. The additional experiments and validation during rebuttal further strengthen the evidence for these contributions and our main findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.