- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes AICORE, an LLM agent that optimizes C/C++ programs by combining source rewrites, loop pragmas and compiler parameters. An agent-facing layer turns Clang remarks, pass traces and IR excerpts into short reports for a selected code region. Each candidate is compiled, checked against the reference output and timed, and is kept only if it is valid and faster. On 30 PolyBench kernels and 29 cBench programs, AICORE reaches 4.49× and 1.30× arithmetic-mean speedup over -O3, with all final outputs matching. OpenCode with the same LLM gets higher means (6.33×, 2.20×) but fails the output check on 7 and 4 programs.
Critical strengths
- Combining source and compiler actions in one loop is a reasonable idea. The doitgen and bicg examples clearly show how remarks about missed hoisting or vectorization opportunities guide source changes that enable vectorization.
- The PolyBench ablations are consistent with the combined design. The full system beats rewrite-only (3.01×), no-compiler-evidence (3.50×) and parameter-only (1.03×).
Critical weaknesses
- The large speedups and baseline strength are unclear. Five PolyBench kernels above 10× drive the arithmetic mean, but the transformations behind these gains are not explained. The fidelity of the AutoPass reimplementation is unclear, and close PolyBench systems (HintPilot, ComPilot, T-LLM Compiler) are not compared.
- Validation and performance stability need clarification. If the ablations follow Algorithm 1 with identical checks during search and final evaluation, their 96.67% and 73.33% pass rates need explanation. OpenCode's check-and-revert policy is also unclear. Independent-search variability and validation on held-out inputs are not reported. I believe that the experiment should also carry out more statistical analysis, such as significance tests for the speedup and correctness claims to provide p-values and effect sizes like Cohen's d.
- I believe the evaluation can be stronger if it included a more comprehensive comparison with existing systems or tools and a more detailed qualitative analysis of the underlying causes of the observed performance improvements. Also, the evaluation covers two benchmark suites using one hardware/compiler configuration and one LLM. It feels hard to judge if the gains extend to larger real-world applications or other platforms.
Questions for authors
- Which transformations produce the >10× PolyBench speedups, and which dataset size was used? Do these transformations introduce multiple threads, external libraries, or changes to floating-point semantics?
- What are the geometric-mean speedups on both suites? How did you verify that the AutoPass reimplementation faithfully reproduces the original method?
- Do the ablation variants and OpenCode use the same validation checks and retention rule as AICORE? If so, why do the ablation variants have pass rates below 100%? What numeric tolerance is used?
- Were the results evaluated across independent optimization searches, and were the retained candidates checked on held-out inputs or problem sizes? If so, please report the variability and validation results.
3 comments
Response to Reviewer (2/2)
Q4: Search Trajectory Evolution and Multi-Scale Generalization
Speedups progress monotonically across the 9-round search loop from initial modifications to final convergence, and robustly scale up to 29× on held-out larger inputs with 100% correctness verification.
1. Multi-Turn Search Trajectory Evolution (Progression across 9 Rounds)
| Kernel | Initial (Round 1) | Intermediate Round (Rewrite/Pragma) | Final Confirmed (Table 1) | Trajectory Progression |
|---|---|---|---|---|
trmm | 1.00× | 12.92× (4-row blocking rewrite) | 17.13× | 1.00× → 12.92× → 17.13× |
covariance | 1.00× | 3.85× (L1 buffer tiling) | 16.37× | 1.00× → 3.85× → 16.37× |
gramschmidt | 1.00× | 6.59× (Loop fission) | 14.63× | 1.00× → 6.59× → 14.63× |
correlation | 1.00× | 9.70× (3D tiling) 14.53× (SCEV tuning) | 13.61× | 1.00× → 9.70× → 13.61× |
symm | 1.00× | 4.16× (Unit-stride rewrite) | 10.24× | 1.00× → 4.16× → 10.24× |
2mm | 1.00× | 5.10× (2D register tiling) | 9.82× | 1.00× → 5.10× → 9.82× |
3mm | 1.00× | 5.13× (Loop fusion & tiling) | 6.56× | 1.00× → 5.13× → 6.56× |
2. Generalization across Problem Sizes (100% Numerical Pass Rate)
| Kernel | MINI* | SMALL* | MEDIUM | LARGE (Optimization) | EXTRALARGE (Held-Out) | Verification |
|---|---|---|---|---|---|---|
correlation | ~1.00×* | 1.21×* | 2.62× | 11.79× | 29.31× | 100% Pass |
covariance | ~1.00×* | 1.24×* | 3.85× | 11.88× | 27.72× | 100% Pass |
syr2k | ~1.00×* | ~1.00×* | 1.00× | 8.65× | 14.45× | 100% Pass |
trmm | ~1.00×* | 1.46×* | 4.43× | 6.40× | 13.38× | 100% Pass |
gramschmidt | ~1.00×* | ~1.00×* | 3.14× | 6.59× | 10.39× | 100% Pass |
2mm | ~1.00×* | ~1.00×* | 2.49× | 5.10× | 7.03× | 100% Pass |
symm | ~1.00×* | 1.40×* | 6.43× | 4.36× | 4.66× | 100% Pass |
*Note on MINI & SMALL: Execution times are sub-millisecond as small matrices fit entirely inside CPU L1/L2 caches. While actual throughput is near or slightly above 1.00×, wall-clock timing jitter at microsecond scale prevents exact speedup profiling. Once problem dimensions expand into memory-bound regimes on held-out EXTRALARGE, AICORE's cache tiling and SIMD vectorization pay off exponentially, scaling up to 10×–29× with 100% verified output correctness.
Additional Points Raised in the Weaknesses: Practical Utility & Platform Roadmap
AICORE's gains reflect genuine, practical performance improvements by eliminating universal hardware-level memory and SIMD bottlenecks with 100% numerical correctness on core workloads, and cross-platform evaluation (e.g., x86-64) will be added in future work.
Follow-up before the discussion period ends
Dear Reviewer njBX,
The author-reviewer discussion ends on November 18 (AoE), and revisions can be uploaded until November 19. If you can spare a few minutes, please let us know whether our comment addresses the points you raised in your review. If a concern remains, a short note is enough, and we will answer before the discussion closes.
Thank you for your time.
Response to Reviewer (1/2)
Thank you for your valuable feedback. We address your questions and concerns below:
Q1: Mechanisms Behind >10× Speedups, Dataset Size, and Execution Constraints
The five >10× gains result from compiler-guided AST transformations eliminating memory vectorization blockers under standard single-threaded C execution, with zero multi-threading, zero external/BLAS libraries, and zero floating-point modifications.
trmm4.50×non-consecutive or strided memory access on A[k][i]; unroll cost exceeds threshold-mllvm -partial-unrolling-threshold=15017.13×covariance1.00×cannot prove pointer independence (aliasing); severe cache miss overheadBLOCK=32); loop permutation(ib,k,i,j);__restrict;-partial-unrolling-threshold=20016.37×gramschmidt1.00×loop-carried scalar dependency cycle prevents vectorizationrestrict;#pragma clang loop vectorize(assume_safety)14.63×correlation10.50×ScalarEvolution check complexity (24) exceeds threshold (16)--pragma-vectorize-scev-check-threshold=25613.61×symm4.16×high-cost predicated strided memory write stallstemp2[_PB_N]; unit-stride accumulation;-mllvm -vectorize-num-stores-pred=1610.24×All speedups are evaluated under the standard PolyBench/C 4.2.1
LARGE_DATASET(verified across all 5 official sizes:MINI,SMALL,MEDIUM,LARGE,EXTRALARGE), strictly using pure single-threaded C execution with zero multi-threading, zero external/BLAS libraries, and zero floating-point semantic modifications (no-ffast-math).Q2: Geometric-Mean Speedups & AutoPass Faithful Reproduction
AICORE achieves superior geometric-mean speedups with 100% output correctness over OpenCode across both suites, and our reproduced AutoPass strictly matches the original paper's published performance range.
2.68×2.91×1.02×1.009×–1.040×1.17×1.26×1.01×1.009×–1.040×Across both suites, AICORE achieves geometric-mean (GM) speedups of
2.68×on PolyBench and1.17×on cBench with100%output correctness, compared to OpenCode's Table 1 GM of2.91×and1.26×which achieves only a76.7%pass rate due to unverified outputs and compilation failures. Following its original specification, we faithfully reproduce AutoPass via a budgeted 3-round search loop driving the 4-agent (Score–Analysis–Reasoning–Evaluation) pipeline with rollback; our reproduced geometric-mean speedups strictly match the original paper's reported performance range (Table 6:1.009×–1.040×).Q3: Ablation Validation, Retention Policy, and Numerical Tolerance
All ablation variants share identical validation checks and monotonic retention rules against the
-O3baseline with 100% numerical correctness; pass rates below 100% strictly reflect the success rate in finding gains over-O3rather than calculation errors.All ablation variants share AICORE's exact validation and monotonic retention rules, verifying outputs against the
-O3baseline under strict element-wise tolerance (max(∣ri∣,∣r^i∣,1.0)∣ri−r^i∣≤10−4 for floating-point, bit-identical for discrete). While all variants maintain 100% numerical correctness without accepting any invalid candidates, Table 2's<100%pass rate strictly reflects the optimization success rate finding gains over-O3(where variants safely retain the 1.00× baseline whenever parameter tuning alone cannot beat-O3), unlike OpenCode whose lower pass rate stems from actual validation failures.