CONTEXTBENCH: A Benchmark for Context Retrieval in Coding Agents

LLM-based coding agents have shown strong performance on automated issue resolution benchmarks, yet existing evaluations largely focus on final task success, providing limited insight into how agents retrieve and use code context during problem solving. We introduce CONTEXTBENCH, a process-oriented evaluation of context retrieval in coding agents. CONTEXTBENCH consists of 1,136 issue- resolution tasks from 66 repositories across eight programming languages, each augmented with human-annotated gold contexts. CONTEXTBENCH implements an automated evaluation framework that tracks agent trajectories and measures context recall, precision, and efficiency throughout issue resolution. Comprehensive experiments on CONTEXTBENCH across seven frontier LLMs and seven coding agents reveal that simple shell-based agents achieve the strongest overall context retrieval, while specialized retrieval mechanisms provide no consistent gains, echoing the “Bitter Lesson” for coding-agent design. Most evaluated LLMs show higher recall than precision at the block and line levels, and substantial gaps exist between explored and utilized context. CONTEXTBENCH augments existing end-to-end benchmarks with intermediate gold-context metrics that unbox the issue-resolution process. These contexts offer valuable intermediate signals for guiding LLM reasoning in software tasks