- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In the paper, the authors introduce a framework named “Benchscope”, designed for understanding benchmark composition and performance differences per system without needing to specify a capability taxonomy beforehand. The paper structures in separating the analysis into two stages: (1) discovering the semantic factors from the benchmark and then (2) testing whether the discovered semantic factors explain the behavior of the system.
First, the paper uses a frozen embedding model to encode the benchmark items. Then, a sparse autoencoder (SAE) discovers recurring semantic factors in each benchmark and go through cross-seed matching and stability thresholds to determine which factors are retained. Here, the LLM-generated descriptions provide an interpretation of the discovered factors. The retained semantic factors are the fixed interaction dimensions in an item-response model (IRT). The model combins overall system performance, an item-difficulty predictor and specialization (per system) along the retained factors.
The authors conduct an evaluation spanning seven coding and agent benchmarks, in which the descriptions of the factors are evaluated through the activation prediction on disjoint items. The paper shows that the AppWorld benchmark provides the strongest generalization result and other tested benchmarks show substantially weaker agreement. Overall, the paper shows that the benchmark and system profiles illustrate differences obscured by broad task labels and aggregate scores.
Critical strengths
- The two-stage process of factor discovery and behavioral validation is well motivated in the paper. I find the decision to learn semantic axes without system responses particularly useful. It avoids defining a dimension from performance patterns and then treating those same patterns as evidence that the dimension matters. The authors also go beyond producing plausible factor labels: held-out prediction and difficulty-matched permutation tests assess whether the semantic assignments contain behavioral signal. This makes the framework more convincing than a purely descriptive benchmark-clustering analysis.
- Also, the evaluation scheme considers the reproducibility of semantic factors well. The paper checks cross-seed activation agreement and top-item overlap. Appendix B also selects the embedding and SAE configuration on a separate, non-Verified corpus without behavioral supervision. Overall, the paper shows some meaningful evidence that the approach can recover recurring structure.
Critical weaknesses
- The description showing that the semantic factors are interpreted as system capability demands is not convincing. In the paper, the experiments show that semantic factors predict system-dependent outcomes, but they do not distinguish task demands from correlated attributes. Despite the reported tests support behavioral association, I am less convinced that the resulting axes lead to the interpretation of stronger capability.
- The behavioral gains shown in the paper are rather modest. Table 2 shows that outside AppWorld, other benchmarks have low correlations and pairwise ranking accuracy, suggesting that the reliability of individual system profiles remains unclear (perhaps having uncertainty estimates might help). Moreover, the main comparisons are against a no-interaction baseline and permuted factors; neither tests whether SAE outperform other alternative baselines like embedding clusters, using PCA features, or embedding-based interactions. This limits my confidence that the framework identifies uniquely informative demands.
Questions for authors
- Do the predictive gains and specialization patterns remain even after accounting for the repository and programming language? Also for Multi-SWE-bench, how can we ensure that the profiles are not primarily driven by language-specific evaluation coverage? Perhaps a within-language or common-evaluation-subset analysis would directly address this concern.
- As mentioned in one of the weakness, have the authors compared Benchscope with similarly interaction models using embedding clusters, PCA features, or the metadata of similar dimensionality?
- Is the separation illustrated in Figure 4 statistically significant? Also, do they persist across item splits and SAE seeds tested in the experiments?
- I am curious what specific criterion did the authors use to identify an individual factor as "behaviorally grounded"?
- Is the "separate LLM annotator" a different model from the description generator? If so, could you specify the LLM used, prompts, number of description examples, and evaluation-item sampling procedure?
- What explains the different results for the apparently identical Qwen3–BatchTopK configuration in Tables 4 and 6, for example, 33 versus 34 stable factors and 96.8% versus 98.3% coverage?
Thanks for the rebuttal, I have raised to a weak acceptance. Id recommend for the final draft having more emphasis on interpretation.