acceptodds
Under review as a conference paper at ICLR 2027

SAR-Arena: Benchmarking Agent-Driven Research on AI Computing Systems and Architecture

Abstract

Research agents are starting to tackle systems and architecture problems as AI workloads and computing platforms evolve rapidly. Such work begins with workload-grounded bottleneck analysis, couples cross-stack mechanism design with engineering implementation, and requires long-horizon, measurement-driven optimization of quantitative system objectives; yet aggregate implementation scores reveal little about how agents make these decisions. We introduce SAR-Arena, a benchmark of ten recent, underexplored workload–platform problems for evaluating research in AI computing systems and architecture. Each task fixes its workload distribution, quality-constrained objective, and baseline while leaving the optimization mechanism open, and offers three expert-designed settings that progressively supply a bottleneck analysis or method proposal. SAR-Arena pairs rubric-based judgments of submitted diagnoses and methods with task-specific quantitative evaluators of runnable solutions, so intermediate reasoning and final system behavior can be examined together. We evaluate six model–harness configurations in 180 runs with eight-hour engineering budgets and extend 18 full-cycle trajectories to 24 engineering hours. The main experiment finds that strong diagnoses and proposals do not consistently yield baseline-beating implementations; relative to full-cycle research, the expert-proposal setting has higher mean engineering scores chiefly through gains on model–task pairs that already beat baseline in both settings. Longer runs yield task-dependent gains, and a nine-category trajectory audit documents recurring measurement, mechanism, and iteration failures, showing that SAR-Arena can reveal where plausible research plans stop short of validated improvement.

Primary area:
datasets and benchmarks
Keywords:
research agents, ML systems, hardware–software co-design, scientific discovery, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_y4m5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes SAR-Arena, a benchmark consisting of 10 systems and architecture optimization tasks. Each task can be evaluated in three different modes: full cycle research, guided research, and engineering optimization. The difference between the three modes is the amount of expert information that is given for each subtask that the agent is responsible for (“bottleneck analysis”, “method design”, and “engineering iteration”).

Critical strengths
  • Tasks are timely and important for the current generation of LLMs to improve upon, benchmarked models and harnesses are relevant and practical.
Critical weaknesses
  • The writing is very unclear, not concise, and hard to follow. It is riddled with odd phrasing (e.g. "stage-end") and it is not clear why the tasks are structured and scored the way that they are (e.g. in line 252, why are those specific weights chosen?).
Questions for authors
  • What exactly distinguishes this benchmark from existing ones and what exactly are the findings and takeaways from the failure analysis?
Reviewer_fAEYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

SAR-Arena is a benchmark of ten AI systems and architecture research tasks, from streaming video understanding and coding-agent serving to retrieval sparse attention. Each task fixes a workload, a hardware platform, a quality-constrained objective and the strongest pre-measured baseline, and evaluates how well an AI agent optimizes each task. Three research settings vary what the agent receives: Full-Cycle Research (the agent does bottleneck analysis, method design and engineering), Guided Research (it receives an expert bottleneck analysis) and Engineering Optimization (it receives an expert method proposal). Diagnosis and method reports are graded by GPT-5.6 Pro against binary rubrics, and implementations by task-specific evaluators, with an engineering score scaled from the baseline to the best evaluated result. Across 180 runs of six model–harness configurations with eight-hour engineering budgets, Claude Opus 5 leads Full-Cycle and Guided Research and GPT-5.6-Sol leads Engineering Optimization, expert input raises engineering scores, and extending 18 runs to 24 hours gives a median 3.0% further gain.

Critical strengths

(+++) The tasks are well-defined research problems. Each fixes a workload, platform and end-to-end objective that charges the full cost, and validity gates block shortcuts.

(++) Comparing three levels of expert input on the same tasks gives an interesting finding: agents implementing an expert's proposal score higher than agents working from their own diagnosis on all tasks, which suggests that their ability to choose what to build is weaker than their ability to build it.

Critical weaknesses

(---) Diagnosis and method scores come from an LLM judge, and their agreement with human experts is never validated. So these scores, 40% of the Full-Cycle total, cannot be reproduced by later studies if the LLM used in this paper is deprecated.

(--) Judged scores may not reflect whether a plan works: DeepSeek-V4-Flash receives up to 45.8 for its reports while its implementations never beat the baseline.

(--) The engineering score is normalized so that the best evaluated run scores 100, which is a reference that changes as new agents are added. Normalizing by the expert solution's gain, as a fixed reference, would keep scores stable as new agents are added.

(--) Each model–task–setting combination is run once, so differences between configurations and settings have no estimate of the variance.

(--) Test-time scaling only extends single runs to 24 hours on three tasks, with no comparison against parallel runs at the same budget, which is a common form of test-time scaling.

(-) Nine of the ten tasks run on a single consumer RTX 5090 instead of datacenter GPUs such as the H100, so the proposed benchmark is not directly usable or comparable on common datacenter GPUs.

(-) The writing is dense and hard to parse, with long noun phrases and many items packed into single sentences.

Questions for authors
  • Can you report the engineering results of the expert solutions, so that agent scores can be read against an expert reference?
  • As the paper itself notes, independent expert rescoring of a sample of reports is still needed. Reporting the judge's agreement with expert raters would make the diagnosis and method scores interpretable.
  • Do the diagnosis and method scores change with a judge from a different model family? GPT-5.6 Pro shares a family with one of the evaluated agents.
  • How does extending one run to 24 hours compare with three independent eight-hour runs at the same budget?
Reviewer_xVn5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SAR-Arena evaluates AI agents on systems and architecture research tasks, separating bottleneck diagnosis, method design, and implementation under three levels of expert guidance. It combines rubric-based judgments of research reports with task-specific performance and validity checks. Experiments compare several agent configurations, examine longer optimization runs, and characterize observed failures.

Critical strengths

Relevant motivation. Evaluating whether agents can connect measurements, design decisions, and working implementations addresses an important gap. Final performance alone provides limited insight into where a research workflow succeeds or fails.

Meaningful systems perspective. The emphasis on complete transactions, quality constraints, and resource accounting is appropriate. These requirements can distinguish useful system improvements from local speedups that merely move costs elsewhere.

Critical weaknesses

The technical contribution remains insufficiently established. The combination of research stages, expert guidance, and heterogeneous workloads is interesting, but the paper needs to identify the new evaluation capability it enables and the difficult technical problems solved in implementing it. Detailed task requirements demonstrate awareness of challenges; they provide limited evidence of how those challenges were resolved.

The reported improvements and research-stage scores need stronger validation. The paper lacks concrete agent case studies connecting a measured bottleneck, a substantive mechanism, and a verified end-to-end gain over a strong baseline. Provisional evaluator documentation, incomplete scoring reconciliation, and limited independent validation of report judgments further weaken confidence in the central measurements.

Questions for authors

What specific capability does SAR-Arena measure that existing executable research and systems-optimization benchmarks cannot? Please distinguish the methodological contribution from expanding workload coverage.

Please choose one task and explain the hardest evaluation problem you solved. For example, how does your evaluator catch a solution that appears faster because it skips work or produces worse outputs?

Please walk through one successful example: what was the bottleneck, what did the agent change, and how much did the complete system improve while preserving quality? Could ordinary parameter tuning achieve a similar gain?

Can the reported diagnosis and method scores be reconstructed from criterion-level judgments? What independent expert evidence supports their reliability, particularly for novelty and valid solutions that differ from the expert reference?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.