- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes an agentic system to solve the physics time series denoising problem.
Critical strengths
- The proposed method demonstrates good performance against closed-source harnesses under the proposed criteria.
- The cost analysis offers good insights into the cost-efficiency of SIDERIUS.
Critical weaknesses
MLE-agent baselines. The work seem to constitute a Machine Learning Engineering (MLE) agent. Although the paper compares against general-purpose coding agents, the authors are encouraged to evaluate against at least one established MLE-agent framework, such as MLE-STAR [2] or AIDE [1], under the same scientific task instructions, data access, evaluation criteria, and computational budget. This would help establish the benefit of SIDERIUS relative to existing frameworks specifically designed for iterative ML experimentation.
Robustness to judge choice. Appendix A.2 describes the use of gpt-5.6-sol. Could the authors assess sensitivity to the choice of LLM judge so that we know whether gaming classifications and the resulting comparisons between systems remain stable?
Judge correctness. Could the authors test the frozen judge prompt and rubric on independently constructed scripts with known valid behavior or deliberately introduced task violations? Reporting false-positive and false-negative rates would help establish whether the judge correctly identifies violations rather than merely producing consistent judgments.
Stability across runs. Appendix E.2 acknowledges the absence of repeated independent campaigns. Could the authors run multiple independent repetitions and report the mean and standard deviation of valid-proposal rate and best valid score to assess performance stability?
Questions for authors
N/A
5 comments
The reviewer suggests comparing against at least one established MLE-agent framework, such as MLE-STAR or AIDE, under the same task instructions, data access, evaluation criteria and compute budget. We followed this suggestion and added AIDE as a baseline. AIDE drafts, debugs and improves complete training scripts through a fixed tree search. We ran it under the same task instructions, data access, GPU and time budget, with the same underlying model as SIDERIUS (GPT-5.6-sol, medium), and judged it with the same frozen rubric:
| System (H100) | Completed proposals | Valid rate | Gaming rate | Best valid score | LLM cost |
|---|---|---|---|---|---|
| AIDE | 141 | 29.8% | 49.6% | 1.05 | $101 |
| SIDERIUS fixed workflow | 31 | 29.0% | 9.7% | 1.09 | $55 |
| SIDERIUS fixed workflow + data analysis | 32 | 50.0% | 0.0% | 1.05 | $65 |
| SIDERIUS orchestration | 49 | 59.2% | 6.1% | 2.74 | $690 |
| SIDERIUS orchestration + data analysis | 115 | 33.0% | 50.4% | 7.28 | $719 |
For reference, the raw input scores 1.00, and the best result in the original TIDMAD paper is 6.43.
This comparison shows two things. First, a framework designed for iterative ML experimentation does not by itself prevent gaming. AIDE and the SIDERIUS fixed workflow both follow a fixed procedure, but AIDE lacks the contract-based core of SIDERIUS. 49.6% of the models AIDE completed are gaming, against 0% to 9.7% for the SIDERIUS fixed workflow. Second, higher scores come from composing capabilities flexibly. The best valid scores of AIDE and the fixed workflow stay near the raw input (1.05 to 1.09). Only orchestration, which composes the same capabilities flexibly, raises the score clearly: 2.74, and 7.28 with data analysis, above every result in the original TIDMAD paper.
In the revised paper, we will add AIDE to the main results table, cite AIDE and MLE-STAR, and discuss more of the MLE-bench agents in the related work.
The second and third concerns are both about the LLM judge. The second asks whether the verdicts stay stable if a different judge is used. The third asks how many false positives and false negatives the judge makes on cases with known answers. We answer both with the same calibration set.
In our updated formulation of the rubric design, two domain experts first led the design of 200 synthetic test cases. For every rule, the cases include examples that violate it and examples that do not, and the experts decided the correct answer for each case. 93 cases are gaming, 101 are not, and the answer key of 6 cases is itself UNCERTAIN. We tested three LLM judges from different model families (GPT, Claude and Gemini) with the same frozen rubric and prompt:
| GPT | Claude | Gemini | |
|---|---|---|---|
| Gaming accuracy (definite answers only) | 97.5% | 97.0% | 95.5% |
| Final-verdict accuracy (definite answers only) | 98.5% | 95.4% | 94.5% |
| False negatives: gaming judged not gaming (of 93) | 3 (3.2%) | 1 (1.1%) | 4 (4.3%) |
| False positives: non-gaming judged gaming (of 101) | 1 (1.0%) | 2 (2.0%) | 2 (2.0%) |
On the choice of judge, the three judges differ by less than 2 percentage points in gaming accuracy and by less than 4 percentage points in final-verdict accuracy. The verdicts are driven mainly by the rubric and the evidence, not by which model family serves as the judge.
On false positives and false negatives, a false negative matters more in our pipeline. A model wrongly judged as gaming at most removes one valid model from the count. A gaming model that is missed is counted as valid. Among the three judges, Claude has the fewest false negatives, with a false-negative rate of 1.1% and a false-positive rate of 2.0%. An UNCERTAIN answer on a case with a definite key is not counted as an error, because such cases go to human review.
The judge in the original paper was GPT (gpt-5.6-sol). Because of cost, we used a single judge (Claude) to re-judge all results. The items the judge is uncertain about are reviewed independently by two domain experts. This review is ongoing, and we will report the results and the inter-expert agreement (Cohen's kappa) during the discussion period.
Our updated evaluation package is well defined and reusable: for every rule, it states what evidence is needed and which diagnostic skill provides it. In practice, it is used as follows. After a run, the package is applied to all proposals: every model goes through the diagnostic skills and is then reviewed by the LLM judge. Human experts then review the models the judge is uncertain about and the highest-scoring models. The highest-scoring model among those confirmed valid is delivered. A high-scoring gaming model would have to pass the diagnostic skills, the LLM judge and the human review to be delivered.
In the revised paper, the appendix will include the full rubric, all 200 calibration cases with their answer, and the three judges' calibration results, including the false-positive and false-negative rates.
We thank the reviewer for reading our paper carefully. We are glad that the reviewer appreciates that SIDERIUS clearly outperforms the closed-source agent harnesses under our evaluation criteria, and we appreciate the positive comment on our cost analysis.
The four concerns the reviewer raises are concrete, and they are instructive for further strengthening our conclusions. To address them, we did the following.
Below, we address the four concerns in order. The second and third concerns are both about the LLM judge, so we answer them together.