- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CHICO, a repository-level code optimization system that combines profiling with lightweight static analysis to select hotspot and caller candidates. Caller candidates receive reuse or reorganization suggestions, which guide a SemOpt-based optimizer. The evaluation compares complete systems on GSO and SWE-fficiency-Lite across four language models and includes a controlled comparison of localization inputs.
Critical strengths
- The opportunity-based caller-selection design is worth investigating. Combining loop and call-input checks with execution cost and opportunity type provides a concrete way to select callers for inspection. Supplying both a starting location and an optimization direction is a useful design choice (Sections 2.2–2.3).
- The complete-system results are encouraging. Table 2 reports consistent improvements in optimization metrics across the evaluated models and benchmarks. For example, GSO Opt@1 with Sol increases from 46.08% for POLO to 63.73% for CHICO. Table 3 also provides evidence that the combined localization input helps the optimizer.
Critical weaknesses
- The localization ablation leaves the source of the gains unclear. Table 3 keeps the optimizer, strategy library, validation, and control flow fixed, which is useful. However, it replaces a single Profile-Top1 target with up to three opportunities, adding caller candidates, optimization directions, and supporting evidence together. This establishes the value of the combined input but does not separate opportunity-aware selection from broader candidate coverage or additional guidance. Since opportunity-based caller selection is the central methodological contribution, this uncertainty materially limits how much the experiment establishes about that contribution.
- The contribution beyond closely related localization work is not sufficiently established. EffiHolmes already studies the gap between profiling hotspots and fix locations using execution paths and guided reasoning. CHICO cites it but distinguishes the two mainly by their inputs: a reported performance issue versus a performance test. That distinction is meaningful, and a direct benchmark comparison may not be appropriate. However, I would like a clearer account of the localization capability or efficiency trade-off that CHICO adds. CHICO reports downstream optimization success without directly analyzing the selected locations. Together with the limited comparison in Section 4, this leaves the methodological advance less clear than the complete-system improvement. This affects my assessment of originality and significance.
Questions for authors
- Clarification of contribution attribution. Do you have comparisons or existing run-level analyses that separate the benefit of opportunity-aware selection from providing additional caller candidates and optimization guidance? In particular, does the advantage persist against cost-ranked hotspot/caller candidates under comparable candidate and optimization budgets?
- Clarification of the ranking criterion and benchmark coverage. Have you examined the relationship between call multiplicity, average per-call cost, and successful caller-side edits in the two benchmarks? What evidence supports the current cumulative-cost and opportunity-type selection rule as a way to prioritize useful candidates? Frequent calls need not be redundant, so this question concerns avoidable work, not call count alone.
- Clarification of the relationship to EffiHolmes. Beyond input requirements, what localization capability or efficiency trade-off distinguishes CHICO, and which existing results or concrete cases support that distinction? Evidence need not take the form of agreement with a single expert patch, since multiple edit locations may yield valid optimizations.