- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
AIS-Bench constructs aligned Context–Idea–Code–Paper quadruplets from published machine-learning research. The benchmark separately evaluates reference-conditioned idea reconstruction, implementation of a supplied reference idea, and report writing from controlled evidence. Thirteen configurations span prompting strategies, specialist systems, and adapted end-to-end agents. The reported findings identify implementation fidelity as a bottleneck and document functional-logic failures, exaggerated results, and unresolved citations
Critical strengths
- A substantial corpus with aligned research artifacts and phase-specific interfaces; the tasks measure reconstruction and fidelity more directly than unconstrained scientific discovery.
- Controlled localization of failures supported by separating implementation and reporting failures from upstream idea-reconstruction errors, while auditing beyond execution targets meaningful scientific deviations.
Critical weaknesses
- Possible mismatch between supplied information and evaluation demands. Distilled ideas omit implementation details and hyperparameters, yet code is compared with reference symbols and numerical trajectories; the sufficiency of the coding specification is unclear.
- The implementation-bottleneck claim depends on a common human-validity analysis, but its decisive numerical results and details are not reported in the supplied PDF.
Questions for authors
- Beyond the distilled idea, exactly what architecture, interface, initialization, optimizer, and hyperparameter information does the coding agent receive or have fixed by the harness? Which reference properties remain unspecified?
- What are the common human-validity rubric, matched sample counts, phase scores, paired differences, and confidence intervals underlying the implementation-bottleneck claim?
2 comments
Thank you for your reply, I have increased my score
We thank the reviewer for distinguishing agent-visible information from harness controls. Our specification ablation directly examines the importance of this distinction.
1. Coding specification and evaluation conditions
Phase II supplies the reference Motivation, Mechanism, and Outcome, not a complete implementation specification. Appendix H.1 retains algorithmic logic but excludes programming syntax, library/hardware details, and training hyperparameters.
The information boundary is as follows:
Harness controls do not reveal every reference choice. Supplying the reference idea removes upstream generation errors, but not ambiguity within that idea.
We tested 100 matched tasks under three information conditions, totaling 300 runs:
Code–Idea Alignment (CIA) and Dynamic Fidelity use a 0–100 scale. C2 − C0 differences [paired 95% CI], using 10,000 bootstrap iterations, were: Pass@1 +10 percentage points [3, 17], CIA +2.0 [0.4, 3.6], and Dynamic Fidelity +9.5 [5.8, 13.2].
Additional specification improved execution success and numerical agreement in this ablation. This supports the reviewer's concern that information completeness affects performance; it does not establish that all residual errors are mechanistic.
Standard attention replacing requested sparse attention can execute while violating the mechanism (Section 4.4). Conversely, a parameter-independent loss constant preserves gradients but can reduce raw trajectory agreement. Execution success and numerical agreement alone do not establish correctness.
Section 3.3 and Appendices H.1/J.4 will distinguish agent inputs from harness controls and report the ablation. Phase II measures implementation fidelity under a defined information interface. Symbol/complexity scores measure reference similarity; Dynamic Fidelity measures short-horizon numerical agreement.
2. Cross-phase evidence and the implementation finding
Native composites measure different constructs and cannot establish a calibrated phase ordering. The intended common human assessment asks whether each artifact fulfills its reconstruction, implementation, or evidence-grounded reporting role.
The pool began with 200 quadruplets across AI Scientist v1, AI Scientist v2, DeepResearcher, and AI-Researcher: 800 candidate artifacts per phase, 2,400 total. Filtering excluded 400 and retained 2,000 artifacts. Two independent ratings each yielded 4,000 ratings. Weighted Cohen's κ was 0.79 and ICC(2,1) was 0.83.
Intervals use 10,000 paired bootstrap iterations with quadruplets as the resampling unit. Both exclude zero in the reported analysis.
The operational rubric, selection/exclusion accounting, and retained pair counts remain unavailable. The 800 candidate pairs per contrast are not retained sample counts. Agreement and confidence intervals cannot fill these gaps or establish a validated phase ordering.
We will revise the abstract and Section 4.2 around the supported finding: mechanism-level failures can survive execution, and specification completeness affects measured performance. Human score differences will remain reported observations, without attributing the entire implementation deficit to mechanism errors.
If these responses address your concerns, we would appreciate reconsideration of your rating. Otherwise, we welcome specific feedback on the remaining concern and evidence needed.