W1. Insufficient motivation and unclear practical significance
Although efficient LLM evaluation is a meaningful research direction, the paper does not sufficiently establish why its particular formulation of general-purpose model profiling addresses an important or pressing practical problem.
A substantial portion of the proposed downstream tasks involves predicting properties such as model license, parameter count, release date, context length, download count, and model family. For open-weight models, most of these attributes can be obtained directly from Hugging Face model cards or configuration files.
It is therefore unclear why practitioners would need to execute hundreds of inference queries to predict information that is already directly accessible.
Furthermore, statistical correlations between model outputs and external metadata do not imply that these metadata attributes are intrinsic or uniquely identifiable from model behavior. For example, download counts and licenses can change without any change to the model's parameters or output distribution.
The paper also combines two fundamentally different categories of prediction targets: behavioral or capability-related properties that may genuinely require evaluation, and external metadata that can be retrieved directly.
W2. Insufficient justification for the information content of a few hundred probe questions
A central claim of the paper is that a few hundred probe questions are sufficient to construct informative model profiles. However, the paper does not adequately investigate whether such limited observations can support reliable, multidimensional, fine-grained capability assessment.
For example, two language models may perform almost identically on the selected 400 general-purpose questions while exhibiting substantially different capabilities in a specialized mathematical domain, such as combinatorics, algebra, or geometric proof.
If the selected questions do not capture these differences, no subsequent embedding or dimensionality-reduction method can reliably distinguish the models based solely on those observations.
From an information-theoretic perspective, for a fixed training prior (D), if the model representation (Z) is constructed from observed responses (X), then the data processing inequality gives:
I(Y; Z | D) ≤ I(Y; X | D)
where (Y) is the target model property.
Encoding and dimensionality reduction cannot create information that is absent from the original observations. Predictions about unobserved capabilities must therefore rely on correlations learned from previously evaluated models, and these correlations may not generalize to new architectures or specialized fine-tuned models.
Although 400 questions might be sufficient for estimating broad aggregate performance, the effective number of observations relevant to any particular fine-grained capability may be extremely small.
The paper does not provide a theoretical characterization of this limitation or sufficiently systematic experiments on fine-grained capabilities, domain-specialized models, or models with atypical capability distributions.
W3. Misalignment between probe selection and the proposed profiling objectives
BULLPEN selects probe questions using Fisher information derived from a two-parameter Item Response Theory (IRT) model.
This procedure primarily identifies questions that discriminate between models with different levels of general ability.
However, distinguishing generally strong and weak models is fundamentally different from identifying specialized capability profiles.
For instance, questions that effectively separate strong and weak mathematical models may not distinguish models with different strengths in algebra, combinatorics, or geometry.
More importantly, questions selected using correctness-based discrimination are not necessarily informative about stylistic preferences, refusal behavior, safety properties, or model provenance.
The paper does not sufficiently justify why a single probe set optimized primarily for general ability discrimination should be appropriate for all 46 downstream tasks.
W4. Potential overinterpretation of sample-efficiency results
The paper emphasizes that BULLPEN retains approximately 91% of its full-pool performance using only 400 questions instead of 24,118.
However, this number represents the ratio between the predictive skill obtained using 400 questions and that obtained using the full question pool. It does not represent 91% prediction accuracy, reliability, or confidence in the resulting model profile.
If the full-pool representation performs poorly on a particular target, retaining 91% of that performance does not establish practical usefulness.
Furthermore, the comparison between 400 and 24,118 questions is primarily conducted on the 104 models that completed the full question set, using a less restrictive evaluation protocol in which the encoder has access to all evaluated models and only the prediction heads are evaluated out of fold.
In contrast, the main strict evaluation involving all 260 models primarily uses the shared 900-question pool.
These differences in evaluation conditions are important and should not be overlooked when interpreting the sample-efficiency claims.
W5. Limited methodological novelty
Several central ideas explored in this paper have already been investigated in previous work.
LLMmap, LLM DNA, and Idiosyncrasies in Large Language Models have examined how model-generated text can reveal model identity, behavioral patterns, or relationships between models.
Similarly, EmbedLLM, LOCUS, and IRT-based approaches have investigated compact model representations derived from limited evaluation data for capability prediction and model selection.
The main methodological contribution of BULLPEN is to combine correctness indicators, question embeddings, and answer embeddings within a unified profiling framework.
Its implementation relies primarily on established techniques, including pretrained BGE encoders, averaging of response embeddings, PCA, PLS-SVD, and ridge or logistic regression.
The strongest-performing answer-only method is essentially the normalized average of pretrained embeddings of model responses.
Methodological simplicity is not inherently problematic. However, when the algorithmic novelty is limited, a paper should establish a particularly important problem or demonstrate substantial practical improvements over existing alternatives.
We thank the reviewer for a balanced and encouraging review, and for noting that the paper reports where correctness-based encoders remain stronger. We ran 8 of our 24 follow-up analyses in response, with numbers in the table at the end of the general response. They clarify what is new beyond the task suite, why the fused code ties the strongest prior method, which labels overlap the probe, and how robust the embedding is.
You noted that the work is closer to a benchmark study, and asked whether the gain comes from the suite rather than the embedding. We agree that the submission reads as a benchmark study, and the camera-ready version will reframe it around the second latent, response style, within an IRT framework. The gain comes from the answers. With tasks and protocol fixed, answer text beats correctness bits on the 20 new tasks at every width (row 6), and the choice of questions matters little (row 13). On behavior, style and safety it also beats the fingerprinting encoders LLMmap and LLM-DNA (row 10). On lineage alone, LLM-DNA ties our method, and we will say so.
On the fused encoder, you are right that it only ties the strongest prior method over all 46 tasks (row 9). Competence tasks dominate that average, and correctness already measures ability well. Where style matters, answer text alone beats JE-IRT and four other prior encoders on the 11 behavior, style and safety tasks (row 10). The fused code keeps most of this style, but full answer text still does better, most clearly on refusal (row 8). We therefore present the fused code as the best compact representation and answer text as the fuller measure of style.
We agree that the data is limited to single-turn QA, and the camera-ready version will restrict its claims accordingly. Multi-turn, tool-use, agentic and non-English settings remain future work. Our closed-model results show why harder settings matter. These models nearly saturate single-turn correctness, so correctness bits group them together while their answers still separate them by style.
You asked us to "state whether [each new task's] labels use similar questions/answers as the probe." We checked every task and will add a per-task table. No label uses the questions or answers that the encoders read, except answer divergence. Labels computed from answers, such as length, hedging and reward-model scores, use separate questions. Capability labels come from held-out questions of the same benchmarks, so they share format but not items. The rest come from public sources such as model cards or from a separate refusal probe. Answer divergence compares a model's answers with other models' answers, so it must read the stored answers. We keep it out of the answer-text controls.
On robustness, we regenerated answers for 19 models under two system prompts, and for 18 of them at temperature 0.7. Temperature does not move a model's position, but a system prompt does. At temperature 0.7, answer text re-identifies 17 of 18 models, against 11 for correctness bits. Under a neutral system prompt it re-identifies 13 of 19, and under a concise one 10 of 19, while correctness bits do better there (row 15). Reference and query answers must therefore share a system prompt, which the camera-ready version will state as a limitation. For closed API models, heads trained only on open models rank their hedging and answer length almost perfectly (row 17).
We thank the reviewer again for the encouraging assessment. These changes make the paper easier to read and its claims more precise.