- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether coding agents can do quantization research on their own, choosing the method and designing the experiments instead of just reproducing a given method. The authors find that a single agent makes several typical mistakes, such as missing prior work and stopping too early, and propose an Explorer-Reviewer system where an independent Reviewer agent checks the research log and decides when the search can stop. Experiments on 2-bit quantization of three LLMs show that the system finds better recipes than a single agent and the reproduced baselines.
Critical strengths
- The paper is well-written and easy to follow.
- The Explorer-Reviewer design is simple and intuitive, and the agent reproduces existing quantization methods faithfully.
Critical weaknesses
- Limited Evaluation. Each configuration is run only once, and the benchmarks used for reporting are also visible to the agents during the search. It is hard to tell whether the improvements are real or just noise.
- Unclear Source of Improvement. The Reviewer brings in a second model and changes when the search stops, but the compute cost is not reported. The gains may simply come from searching longer rather than from better review.
Questions for authors
- How does the system compare with a single agent given the same compute budget?
- Do the baselines like QTIP and YAQA use the same recovery fine-tuning as the agent recipes?
- Missing references. https://arxiv.org/abs/2605.03042, https://arxiv.org/abs/2407.11062 and https://arxiv.org/abs/2609.28614 should be discussed.