acceptodds
Under review as a conference paper at ICLR 2027

CapAtlas: Resolution-Specific Validity of Sparse Behavioral Model Maps

Abstract

Thousands of language-model checkpoints are poorly documented, yet comprehensively benchmarking every model is impractical. Sparse model maps offer a cheaper alternative, but current evaluations collapse map quality into one score and leave their downstream validity unclear. We introduce CapAtlas, a leakage-controlled protocol that learns a capability map from documented references, freezes it, places an unseen model from 16 response distributions, and only then unseals its evaluation profile. We evaluate a 46-model atlas on 70 targets from 43 unseen lineages, using 84,000 held-out response records and 109,480 sealed oracle outcomes for evaluation. Reference supervision improves graded local retrieval from 0.8181 to 0.8609 lineage-macro NDCG@5 (+0.0428; 95% CI [0.0229, 0.0640]; random floor 0.6048), with a positive effect from checkpoint- to architecture-level aggregation. Six calibration summaries reach 0.8611, showing that calibration explains the aggregate score while itemwise geometry contributes selectively by domain. Under a disjoint capability construct, global ranking improves by 0.1691 Kendall's tau; the registered local-retrieval gate identifies a separate neighborhood-calibration target. On 50 prospective targets, a frozen reliability score stratifies placement risk (AUROC 0.731, 95% CI [0.582, 0.866]), and its 80.95%-precision operating point quantifies a 9.05-point gap to the registered 90% standard. CapAtlas turns sparse model mapping into an auditable instrument whose claims are matched to their decision resolution.

Primary area:
datasets and benchmarks
Keywords:
language model evaluation, behavioral fingerprints, model mapping, capability auditing, query-efficient evaluation, reliability calibration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ypWbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CapAtlas evaluates sparse behavioral model mapping under unseen-lineage transfer. It distinguishes global ranking, local retrieval, reliability estimation, and fixed-precision acceptance. The reported improvements over response-space mapping are positive, but calibration-only features achieve comparable results, and the prospective precision criterion is not met.

Critical strengths

1.The frozen, unseen-lineage evaluation design reduces direct target leakage. 

2.Separate evaluation of retrieval and reliability prevents overinterpreting a single mapping score.

Critical weaknesses
  1. The main representation does not consistently outperform simple calibration features. Table 1 reports NDCG@5 of 0.8609 for SoftMap versus 0.8611 for calibration-only features. Table 2 similarly reports 0.8310 versus 0.8372 under the benchmark-disjoint capability construct. These results leave the additional value of itemwise behavioral geometry unclear, despite the greater complexity of the mapping pipeline.

  2. The prospective reliability target is not achieved. The method reaches 80.95% precision at 42% coverage, below the registered 90% requirement. Both oracle-half evaluations also fall below the target. Although AUROC of 0.731 indicates useful risk ranking, the evidence does not establish a reliable fixed-precision acceptance policy.

Questions for authors

1.Which domains show statistically significant improvements from itemwise features beyond calibration-only representations?

2.What practical advantage justifies the additional complexity over calibration-only mapping?

3.Is the failed precision criterion mainly attributable to risk estimation or threshold calibration?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ypWb (1/4): W1/Q2 - Author response

W1/Q2. Methodological contribution and practical value

Author response

Thank you for asking us to clarify both the methodological contribution of the itemwise representation and its practical value relative to calibration-only mapping.

The methodological distinction concerns what information is retained. Calibration pools responses across probes into mean entropy, maximum probability, and four answer-option probabilities. SoftMap instead retains the association between each question and its response distribution, and learns a low-rank mapping from these itemwise patterns to measured reference capabilities. Consequently, two models can have identical calibration summaries while exhibiting different question-specific response patterns; SoftMap can represent this distinction, whereas the six-summary representation cannot. Both methods use reference capability supervision. The additional modeling capacity of SoftMap is therefore its ability to use question-specific behavioral structure beyond aggregate confidence and option preferences.

We directly tested whether this structure contributes to SoftMap’s predictions. Independently permuting the 16 item blocks within each model preserves all six calibration summaries but disrupts cross-model question correspondence. Across 100 fixed permutations with refitting, aligned SoftMap exceeds this control by 0.05144 NDCG@5 (95% CI [0.03175, 0.07338]; Holm-adjusted p=0.000020) and 0.12234 Recall@5 ([0.07484, 0.17499]; Holm-adjusted p=0.00032). Applying a common permutation to all models leaves predictions unchanged to numerical precision. These controls show that question correspondence, rather than the arbitrary ordering of questions, makes a measurable contribution within SoftMap.

This finding addresses a different question from whether SoftMap outperforms calibration. We agree that the latter advantage is not established consistently: calibration matches or exceeds SoftMap’s aggregate NDCG@5 under both O1 and O2. Adding reference-fitted itemwise residual prediction to calibration yields a gain of 0.00789 NDCG@5 on the original panel, but this contrast does not survive multiplicity correction (p=0.310). We therefore distinguish evidence that the representation uses question-specific information from evidence that retaining this information improves prediction over a strong pooled baseline.

The practical contribution of CapAtlas is to connect a new checkpoint’s sparse responses to models with measured capabilities, producing candidates for further comparison. We evaluated this use beyond the original aggregate scores. On 32 completed targets from 24 new training programs, the frozen estimators used the same 16 probes and achieved NDCG@5 of 0.9072 for SoftMap and 0.9036 for calibration; the paired difference was 0.0036 (95% CI [−0.021, 0.024]). In a retrospective follow-up evaluation on these targets, each estimator nominated ten references, used 24 additional items to rerank them, and returned five. Evaluation on 192 disjoint items yielded program-weighted means of 3.510 true peers for SoftMap and 3.542 for calibration, with a paired difference of −0.031 [−0.208, 0.156]. The O3 calibration comparator additionally includes the mean top-two probability margin. Before follow-up, the corresponding means were 3.344 for SoftMap and 3.302 for calibration. The within-policy gains were +0.167 (95% CI [−0.188, 0.573]) and +0.240 [−0.135, 0.635], respectively; neither survived multiplicity correction. The replay therefore quantifies peer recovery under the tested procedure without establishing a benefit from the additional measurement.

These results provide a concrete implementation recommendation: use calibration for coarse placement and measure the candidates when exact peer identity matters. SoftMap provides a way to represent and test question-specific behavioral information, and the alignment experiment establishes its role within the method. The direct baseline and follow-up comparisons determine whether that additional capacity is worth using in the evaluated setting. We have revised the manuscript to make this methodological distinction explicit and to align the recommended implementation with the observed practical results.


AuthorsOpenReview ↗

Response to Reviewer ypWb (2/4): W1/Q2 - Manuscript changes

W1/Q2. Methodological contribution and practical value (continued)

Changes in the revised manuscript

Section 3.2, p. 4 — information retained by the two representations.

Calibration pools responses into mean entropy, maximum probability, and four option probabilities. SoftMap retains which response distribution belongs to which question. Models with identical pooled summaries can therefore have distinct itemwise representations.

Location 1: Appendix U.1, p. 29 — controlled test of question correspondence.

Independently permuting the 16 item blocks within each model preserves mean entropy, maximum probability, and all four option means: each is a symmetric sum over items.

Location 2: Section 5.3, p. 8 — measured contribution of item alignment.

Across 100 fixed permutations with retraining, aligned SoftMap gains 0.05144 NDCG ([0.03175, 0.07338], Holm p=.000020) and 0.12234 Recall ([0.07484, 0.17499], Holm p=.00032). A common permutation of all models leaves predictions unchanged to numerical precision.

Location 3: Introduction, p. 1 — practical implementation and intended output.

For a new checkpoint we recommend starting with calibration: average its entropy, maximum probability and four answer-option probabilities over the same 16 probes. Fit the map from these six features to measured reference profiles on the atlas alone. This yields a coarse ordering and a list of candidates to measure.

The surrounding discussion in Section 5.3 and Appendix U.1 distinguishes the alignment result from incremental superiority over calibration. Section 5.4 (p. 8) explains the role of the Top-5 as follow-up candidates. Appendices V–W (pp. 32–34) report the frozen new-program comparison and the retrospective follow-up evaluation, respectively.


AuthorsOpenReview ↗

Response to Reviewer ypWb (3/4): Q1

Q1. Domain-specific benefits beyond calibration

Author response

Thank you for asking us to identify where itemwise information provides benefits beyond calibration. No domain-level improvement survives multiplicity correction in the current evaluations. We report the positive estimates as exploratory findings and distinguish them from the controlled evidence that SoftMap uses question correspondence.

In the original five-domain comparison (Appendix T), SoftMap shows positive NDCG@5 differences over calibration in biomedical tasks (+0.03998) and general reasoning (+0.01989). Their unadjusted p-values are 0.04543 and 0.04721, respectively, but both become 0.22715 after Holm correction across the five domains. Thus, these estimates identify possible areas of benefit without establishing a domain-specific advantage.

We also tested incremental itemwise information more directly by adding a reference-fitted itemwise residual predictor to calibration (C+I). In the supplementary fixed-panel analysis (Appendix U.1), this addition improves mathematical-domain NDCG@5 by 0.0190, but the contrast does not survive correction (unadjusted p=0.0196; Holm-adjusted p=0.3101). None of the 18 aggregate/domain conditional NDCG comparisons in that analysis survives its correction. The original five-domain tests and the supplementary 18-comparison analysis are reported as separate correction families.

These results clarify the distinction between the methodological contribution and a domain-specific recommendation. The alignment control discussed in our W1/Q2 response establishes that question correspondence contributes to SoftMap’s predictions. Identifying a domain in which SoftMap should be preferred to calibration requires a direct, corrected comparison against calibration; the current domain results do not establish such a preference. We retain the complete domain results, label the positive differences as exploratory, and do not use them to prescribe domain-specific estimator selection.


Changes in the revised manuscript

Location 1: Appendix T, p. 28 — conclusion of the original five-domain comparison.

No domain passes Holm correction.

The table reports all five domains, including their estimated differences, confidence intervals, unadjusted p-values, and Holm-adjusted p-values.

Location 2: Appendix U.1, Table 21, p. 29 — correction scope for the supplementary comparisons.

The first 18 NDCG rows share one correction family; the final two share another.

This table includes the direct SoftMap–calibration comparisons and the incremental C+I–calibration comparisons, preventing an isolated unadjusted result from being interpreted as a confirmed domain-level benefit.

Location 3: Appendix U.1, p. 29 — interpretation of the alignment experiment.

These controls show use of question correspondence, not incremental superiority over C or N.

Here, C denotes calibration-only and N denotes the metadata-plus-calibration nuisance model. Section 5.3 also distinguishes exploratory domain effects from the contribution of item alignment.


AuthorsOpenReview ↗

Response to Reviewer ypWb (4/4): W2/Q3

W2/Q3. Risk estimation versus threshold calibration

Author response

Thank you for asking us to distinguish these possible explanations. We examined discrimination, probability calibration, and threshold transfer separately. The current evidence does not support attributing the failed precision target solely to threshold calibration, nor does it identify the relative contribution of each source of error.

The original threshold was chosen using an empirical 80% development-prefix criterion, whereas the prospective evaluation required 90% precision. It was therefore not a development-validated 90% policy. The frozen rule resolved 17/21 accepted placements in the first cohort and 14/16 on the new programs. Its AUROC was 0.731 and 0.564, respectively, indicating weaker observed discrimination in the new cohort. Development-fitted sigmoid recalibration changed the first external Brier score only slightly, from 0.21490 to 0.21306; this monotonic recalibration does not change the score ordering. Three development-only 90% alternatives abstained on the external evaluations, leaving precision undefined.

In a retrospective threshold scan, the widest first-cohort prefix reaching 90% observed precision resolved 14/15. Applying that cutoff unchanged to the new cohort resolved 4/4 at 12.5% coverage (one-sided 95% exact lower bound 47.3%, assuming independent outcomes). Selecting a cutoff with access to the new labels gave 10/11 at 34.4% coverage. A stricter threshold can reduce coverage sharply; the data still leave the relative contributions of score quality and threshold transfer unresolved (Appendices U–V).

The post-hoc scan evaluates all distinct cutoffs and keeps tied scores together. The widest qualifying prefix on the first cohort has cutoff 0.70365138 and covers 15/50 (30%). On the new cohort the four selected targets belonged to four programs. The threshold chosen with access to the new labels was 0.65217850; its 10/11 is an in-sample descriptive upper envelope for observed coverage at the requested precision on these scores. Both cohorts had already been analyzed before designing this diagnostic. We retained the original frozen-rule result and do not propose either scanned cutoff as a validated policy.


Changes in the revised manuscript

Locations: Section 5.5; Appendix U, Table 24 (p. 31); Appendix V (pp. 32–33).

Revised text (Appendix V, p. 33):

Both cohorts were already analyzed when we designed this diagnostic. It quantifies the precision–coverage tradeoff and leaves the registered result and the source of the transfer error unresolved.

Supporting passage (Appendix V, p. 32):

We retrospectively evaluated every distinct cutoff of the original frozen SoftMap risk score, accepting all ties. On the first 50 targets the widest score prefix with observed precision of at least 90% contained 15 targets, 14 resolved (30% coverage, cutoff 0.70365138, applied at full precision). Applying that same cutoff to the 32 completed new targets accepted four from four programs; all four resolved (12.5% coverage). Even treating those four outcomes as independent Bernoulli trials, their one-sided 95% exact lower bound is 47.3%. Choosing afresh with access to the new labels yields 10/11 at a cutoff of 0.65217850 (34.4% coverage).

Reviewer_G4AAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces CapAtlas, a framework for placing undocumented language models among benchmarked models using responses to 16 multiple-choice probes. Its proposed method, SoftMap, uses regression trained on 46 reference models to predict benchmark scores. The evaluation covers global ranking, top-5 peer retrieval, placement reliability, and fixed-precision acceptance. SoftMap improves NDCG@5 over direct response matching (0.861 vs. 0.818, Table 1), but its advantage over simpler baselines is unclear. Additionally, the reliability-based acceptance rule fails to meet the authors’ 90% precision target (Table 3).

Critical strengths

The paper evaluates model-map quality at different decision levels instead of relying on a single metric, which is useful. Target lineages are excluded from the atlas, and the map and decision threshold are fixed before evaluation. The evaluation covers 70 targets from 43 unseen lineages, with positive gains in 31 of the 43 lineages.

Critical weaknesses

The benefit of the learned geometry of SoftMap is unclear. Calibration-only and nuisance-only controls achieve NDCG@5 scores of 0.8611 and 0.8646, respectively, which are similar to or higher than SoftMap’s 0.8609 (Table 1). Also, Recall@5 is only 0.427, lower than that of other baselines (Appendix G, Table 10). Under the second capability definition, the NDCG@5 gain is only 0.0080, and the confidence interval includes zero (Section 5.4, Table 2). Therefore, it is unclear whether SoftMap performs better than simpler methods.

The acceptance rule does not meet its target. It achieves 80.95% precision at 42% coverage, below the required 90% (Table 3).

The method is sensitive to prompt format. On three targets, chat templates reduce SoftMap’s NDCG@5 from 0.960 to 0.882. Probe selection is also unstable, with a mean Jaccard overlap of only 0.393 across reselections. This raises concerns about how well the method works with different formats and whether the 16-question panel is a reliable choice.

Questions for authors
  • Appendix R describes a “second, larger” prospective cohort of 70 targets from 43 lineages. However, Table 17 reports the same scores and the same 31/43 positive lineages as Table 1, which reports results for the same number of targets and lineages. Are these the same targets? If so, why is Appendix R presented as a prospective replication?
4 comments
AuthorsOpenReview ↗

Response to Reviewer G4AA (1/3): W1

W1. Geometry versus simpler baselines

Author response

Thank you for raising this concern. The practical question addressed by CapAtlas is how to position a new checkpoint among models with measured capabilities when only a small probe budget is available. This positioning gives the probe responses a concrete interpretation: which benchmarked models should be considered for comparison and further measurement? The calibration-only estimator also learns this relationship from the reference capability profiles; its strong performance therefore identifies a simpler implementation of reference-supervised placement. We have revised the manuscript to make this practical contribution explicit and to recommend calibration summaries as the default for coarse placement.

We agree that the evidence does not establish an advantage for SoftMap over the strongest simple controls. On the original panel, calibration-only and nuisance-only achieve NDCG@5 of 0.8611 and 0.8646, compared with 0.8609 for SoftMap. SoftMap’s Recall@5 of approximately 0.427 also shows that a favorable graded ranking does not imply accurate recovery of the exact five nearest peers. Under the second capability definition, the NDCG@5 gain of 0.0080 over raw matching has a confidence interval that includes zero. These results motivate both the simpler default and an explicit distinction between coarse placement and exact peer identification.

To test whether this recommendation extends to new training programs, we compared the frozen estimators on 32 completed targets from 24 new programs, using the same 16 probes. SoftMap and calibration achieved NDCG@5 of 0.9072 and 0.9036, respectively. Their paired difference was +0.0036 (95% CI [−0.021, 0.024]); the Recall@5 difference was +0.0125 [−0.071, 0.092]. Neither comparison establishes SoftMap’s superiority. This additional evaluation supports using the simpler estimator when the immediate objective is coarse placement.

We also clarify how users should act on the resulting shortlist. In a retrospective follow-up evaluation on these same targets, each estimator nominated ten reference models, used 24 follow-up items to rerank them, and returned five. We evaluated these selections using 192 disjoint validation items. At equal follow-up exposure, the program-weighted mean numbers of true peers recovered were 3.510 for SoftMap and 3.542 for calibration; the paired difference was −0.031 peers (95% CI [−0.208, 0.156]). The O3 calibration comparator includes the mean top-two probability margin in addition to the original six summaries. Before follow-up, the corresponding means were 3.344 for SoftMap and 3.302 for calibration. The within-policy gains were +0.167 (95% CI [−0.188, 0.573]) and +0.240 [−0.135, 0.635], respectively; neither survived multiplicity correction. The replay therefore quantifies peer recovery under the tested procedure without establishing a benefit from the additional measurement.

Together, these results provide an actionable use of CapAtlas: use a reference-fitted calibration map to obtain an initial position and candidate set, then measure the candidates when exact peer identity matters. The contribution is a tested procedure for connecting sparse observations to measured reference capabilities, with separate evaluations of graded placement and peer recovery. We have aligned the recommended implementation and the interpretation of the Top-5 with this evidence.


Changes in the revised manuscript

Location 1: Introduction, p. 1 — recommended estimator and intended use.

For a new checkpoint we recommend starting with calibration: average its entropy, maximum probability and four answer-option probabilities over the same 16 probes. Fit the map from these six features to measured reference profiles on the atlas alone. This yields a coarse ordering and a list of candidates to measure.

Location 2: Section 5.4, p. 8 — interpretation of the Top-5 and the follow-up evidence.

The Top-5 are candidates for follow-up. In a separate O3 replay, 24-item follow-up recovered 3.510 versus 3.542 peers for SoftMap and calibration (Appendix W).

The corresponding changes also appear in Algorithm 1, which distinguishes the recommended calibration estimator from the prespecified SoftMap comparison; the practical guidance on p. 10; Appendix V, which reports the frozen new-program comparison; and Appendix W, which describes the follow-up protocol and disjoint validation.


AuthorsOpenReview ↗

Response to Reviewer G4AA (2/3): W2

W2. Precision and coverage

Author response

Thank you for pointing this out. We agree that the acceptance rule did not meet the registered 90% precision requirement. A placement is resolved when at least three of its five returned candidates belong to the oracle Top-5. The original rule accepted 21 of 50 targets and resolved 17 of those acceptances, giving 80.95% precision at 42% coverage.

To test transfer of the acceptance policy, we applied the unchanged frozen rule to 32 completed targets from 24 new training programs. It accepted 16 targets and resolved 14, giving 87.5% precision at 50% coverage. This evaluation also failed the 90% requirement; its one-sided precision lower bound was 65.6%. Moreover, 26 of the 32 targets were resolved before selection, so the observed accepted-group precision does not by itself establish a stable benefit from filtering. The rule was fitted for SoftMap and its results cannot be transferred to the calibration default.

These findings change our usage recommendation. CapAtlas provides candidates for comparison and further measurement, but the current acceptance rule does not justify automatically accepting a placement at the registered 90% precision. When exact peer identity matters, users should measure the candidates. The follow-up evaluation in Appendix W tests this use of the shortlist; it does not substitute for validation of the acceptance policy. We now state the failed precision target explicitly and distinguish candidate prioritization from automatic acceptance.


Changes in the revised manuscript

Abstract, p. 1 — acceptance outcomes and unmet precision target.

Its acceptance rule resolved 17 of 21 accepted placements on the first cohort and 14 of 16 on the second. Neither reached the registered 90% precision.

Section 5.4, p. 8 — intended role of the shortlist.

The Top-5 are candidates for follow-up.

The corresponding acceptance results and usage guidance appear in Section 5.5, the Discussion, and Appendix V; Appendix W separately evaluates follow-up measurement of the candidates.


AuthorsOpenReview ↗

Response to Reviewer G4AA (3/3): W3 and Q

W3. Prompt format and panel selection

Author response

Thank you for raising these concerns. We agree that prompt format and probe selection are part of the measurement protocol and require explicit evaluation. We expanded the prompt comparison beyond the original three targets and added matched-budget comparisons across randomly sampled probe panels.

The prompt analysis uses all 34 atlas checkpoints with paired canonical/native responses, spanning 14 families. Each evaluated family is excluded from regressor fitting, reference scaling, and the candidate atlas. Under canonical reference prompts, native minus canonical target prompting changes SoftMap NDCG@5 by −0.01491 (95% CI [−0.03681, 0.00824]); the corresponding calibration difference is −0.0152 [−0.0419, 0.0107]. With a mixed reference atlas, the differences are −0.0008 [−0.0259, 0.0258] and −0.0100 [−0.0322, 0.0107], respectively. None of the six prompt contrasts passes multiplicity correction. These results broaden the comparison but do not establish template invariance. We retain the original three-target decline. The panel was historically selected using the original atlas, so this is fitting-heldout evaluation, not independent validation of panel selection.

For probe selection, the historical Jaccard overlap of 0.393 remains unchanged. We distinguish reproducibility of the selected question identities from predictive performance with alternative panels. At each budget of 8, 16, 32, and 64 probes, we evaluate 20 fixed-seed, nested, domain-balanced random panels. All methods receive identical items and budgets, and no target outcomes select panels. At the 16-item budget, SoftMap minus calibration is +0.0102 NDCG@5 (Holm p=.2490) and +0.0076 Recall@5 (Holm p=1.0000), averaged over panels. This comparison does not establish an advantage for SoftMap under alternative panels, consistent with recommending the simpler calibration estimator. It does not establish that arbitrary panels are equally reliable: the intervals condition on the sampled panels and omit panel-resampling uncertainty.

The practical contribution is a reference-supervised procedure for connecting sparse responses to measured reference capabilities. The 16-item panel is the evaluated implementation, not a uniquely reliable set. For use of this procedure, we recommend specifying the reference and target prompting protocol in advance and rechecking placement performance when that protocol changes. The additional analyses make the dependence on these measurement choices explicit rather than treating the map as independent of its probing conditions.


Changes in the revised manuscript

Appendix U.3, p. 31 — scope of the prompt-format evaluation.

The fixed panel was historically selected with the original atlas: this is fitting-heldout, not panel-selection-independent validation. None of six prompt contrasts passes correction; the historical three-external-target sensitivity remains relevant.

Appendix U.2, p. 30 — interpretation of the random-panel comparisons.

Lineage intervals condition on the sampled panels and do not include panel-resampling uncertainty. These comparisons quantify predictive variation, not stability of the selected item identities.

Appendix U.2 reports the matched-budget comparisons; Appendix U.3 reports the paired prompt-format comparisons. The Discussion (p. 10) now states the protocol requirement explicitly:

Specify the reference and target prompting protocol before evaluation; changing it requires rechecking placement performance.


Q. Relationship between Table 1 and Appendix R

Author response

Yes. Table 1 and Appendix R use the same 70 targets from 43 lineages. We have consolidated the presentation: the main text reports the results, and Appendix R provides protocol details and supplementary analyses. We have removed the duplicate method-score rows and revised the heading and cross-references to make this relationship clear.


Changes in the revised manuscript

Section 5.1, p. 6 — primary evaluation and supporting appendix.

We report one primary evaluation of 70 targets from 43 atlas-disjoint lineages. Appendix R supplies its protocol details, prespecified contrasts, and supplementary open-set analysis.

Appendix R, p. 25 — scope of the supplementary material.

This appendix provides protocol details and supplementary analyses for the single primary evaluation in Section 5.1 and Table 1: 70 targets from 43 lineages.

Appendix R is now titled “Primary Evaluation: Protocol Details and Open-Set Analysis.” Its supplementary table retains the prespecified contrasts and refers readers to Table 1 for method-level scores.

Thanks for the detailed responses. All of my concerns are not addressed.

  • W1: My main concern still remains the same. A simpler estimator performs similar as SoftMap, so its added complexity is not justified.
  • W2: The new results also miss the 90% target. So, the target is still not met.
  • W3: The prompt analysis is helpful, but format robustness is still not established. So, I am not fully convinced.
  • Q: Thank you for clarifying that Table 1 and Appendix R use the same targets.
Reviewer_G1tpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CapAtlas, a method for placing a previously unseen language model within a capability map using the model’s response distributions on only 16 multiple-choice questions. The reference atlas contains 46 models with known capability profiles, and the evaluation focuses on target models from lineages that are not represented in the atlas. An important aspect of the paper is that it separates several notions of successful placement, recovering the overall capability ordering, retrieving nearby models, recovering the exact Top-5 neighborhood, and determining when a placement is reliable enough to accept automatically.

Critical strengths

The paper’s main strength is its careful evaluation design. It separates overall model ranking, local neighbor retrieval, and placement reliability instead of treating them as the same problem. This is important because the results show that the method can work well for broad ranking while being less reliable for identifying exact neighboring models. Also, the authors clearly report both positive and negative results, including that simple calibration-based models can match SoftMap, local retrieval does not consistently transfer to the second capability definition, and the prospective reliability threshold is not met. This makes the conclusions easier to trust, even though some results are weaker than expected.

Critical weaknesses

The main weakness is that SoftMap does not clearly outperform much simpler baselines. On the primary evaluation, SoftMap achieves an NDCG@5 of 0.8609, while the calibration-only model reaches 0.8611 and the nuisance-feature model reaches 0.8646. This makes it unclear whether the individual 16 questions capture a meaningful capability structure, or whether most of the improvement comes from broad statistics such as model confidence, entropy, and response concentration. The paper presents some domain-level evidence that individual questions can help, but these analyses are retrospective and the improvements are not consistent across domains.

Another weakness is that the method is not yet reliable at the finer resolution needed for its main practical uses. Under the benchmark-disjoint capability definition, SoftMap improves the overall ordering of models, but its improvement in local NDCG@5 is small and uncertain, while Recall@5 remains almost unchanged. This suggests that the method may locate a model in the correct broad capability region without consistently identifying its actual nearest peers. The prospective reliability experiment raises a similar concern. The acceptance rule achieves 80.95% precision, below the paper’s target of 90%, and four of the 21 accepted placements remain unresolved. Therefore, the reliability score may help rank which predictions are more trustworthy, but it is not yet accurate enough to support automatic acceptance. Overall, the results are promising for broad model ranking, but they do not yet establish reliable local-neighbor retrieval or confident certification of individual placements.

Questions for authors

Can the authors clarify what evidence shows that the identities of the 16 individual questions provide useful information beyond aggregate features?

Under the benchmark-disjoint capability definition, SoftMap improves the overall ordering of models but does not clearly improve local NDCG@5 or Recall@5. How should users interpret the returned Top-5 models in this case?

3 comments
AuthorsOpenReview ↗

Response to Reviewer G1tp (1/2): W1/Q1

W1/Q1. Value of sparse placement and the contribution of item identity

Author response

Thank you for asking what the individual questions contribute beyond aggregate response features. CapAtlas uses existing benchmark measurements to guide the assessment of an undocumented model from a small amount of new evidence. On 70 targets from 43 unseen lineages, SoftMap improves NDCG@5 from 0.8181 with direct response matching to 0.8609 using exactly the same 16 probe distributions. This demonstrates that supervision from the reference models’ measured capabilities improves placement under the primary capability definition.

The aggregate-feature controls identify a simpler way to obtain this benefit. Calibration-only achieves 0.8611, and nuisance-only achieves 0.8646. We therefore revised the practical recommendation to use calibration-only for coarse placement. We also evaluated how the resulting candidates could guide further measurement. In a retrospective follow-up on 32 targets from 24 new training programs, each estimator nominated ten references, reranked them using the same 24 follow-up items, and returned five for evaluation on disjoint items. The program-weighted means were 3.510 true peers for SoftMap and 3.542 for calibration, whose frozen O3 variant also includes the mean top-two probability margin. These results support using sparse placement to nominate candidates for further evaluation and favor the simpler implementation.

To address the reviewer’s specific question about item identity, we added two complementary controls. Independently permuting the question blocks within each model preserved all six calibration summaries while disrupting question correspondence across models. Across 100 fixed permutations with retraining, aligned SoftMap exceeded this control by 0.05144 NDCG@5 (95% CI [0.03175, 0.07338]). A common permutation across all models left predictions unchanged. This establishes that SoftMap uses question correspondence.

We then tested whether itemwise information adds predictive value beyond calibration. Adding reference-fitted itemwise residual predictions increased NDCG@5 by 0.00789, but the gain remained uncertain after multiplicity correction (Holm-adjusted p = 0.310). None of the five original domain-specific gains survived correction. We now distinguish the demonstrated value of reference-supervised sparse placement from the unresolved incremental benefit of retaining item-level detail.


Changes in the revised manuscript

Introduction (p. 1). We describe the simpler implementation and its intended use.

“For a new checkpoint we recommend starting with calibration: average its entropy, maximum probability and four answer-option probabilities over the same 16 probes. Fit the map from these six features to measured reference profiles on the atlas alone. This yields a coarse ordering and a list of candidates to measure.”

Section 5.3 (p. 8), with detailed controls in Appendix U.1 (p. 29). We report the conditional test of whether itemwise information improves on calibration.

“Adding itemwise residual prediction to calibration gives +0.00789 NDCG on the original panel, but fails the corrected test (p = .310).”

Section 5.4 (p. 8), with the follow-up protocol and results in Appendix W. We connect candidate nomination to an equal-item follow-up evaluation.

“In a separate O3 replay, 24-item follow-up recovered 3.510 versus 3.542 peers for SoftMap and calibration (Appendix W).”

AuthorsOpenReview ↗

Response to Reviewer G1tp (2/2): W2/Q2

W2/Q2. Interpreting the Top-5 and the reliability of accepted placements

Author response

Thank you for asking how users should interpret the returned Top-5. The shortlist gives an undocumented checkpoint a concrete set of benchmarked reference models for direct comparison. Their measured capability profiles help users interpret the predicted position and choose which candidate comparisons to evaluate next. We have clarified this use in the revised manuscript.

We tested how additional measurement can refine that shortlist. In a retrospective replay on 32 targets from 24 new training programs, each estimator first nominated ten references from the same 16-probe input. We then used 24 follow-up items to rerank the nominees and return five, and evaluated the result on 192 disjoint items. The program-weighted means were 3.510 true peers per returned set for SoftMap and 3.542 for calibration. Before follow-up, the corresponding means were 3.344 for SoftMap and 3.302 for calibration. The within-policy gains were +0.167 (95% CI [−0.188, 0.573]) and +0.240 [−0.135, 0.635], respectively; neither survived multiplicity correction. The replay therefore quantifies peer recovery under the tested procedure without establishing a benefit from the additional measurement. When the reference responses are cached, this step requires 24 additional target responses. This gives users a concrete procedure for moving from sparse placement to a more directly measured peer shortlist.

We also report the reliability attained by the frozen acceptance rule. A placement was defined as resolved when at least three of its five returned references belonged to the oracle Top-5. In the first prospective cohort, the rule accepted 21 placements and resolved 17, achieving 80.95% precision at 42% coverage. Applied unchanged to the 32 new targets, it accepted 16 and resolved 14, achieving 87.5% precision at 50% coverage. These are prospective operating-point results for the SoftMap reliability rule. Both fell below the preregistered 90% precision target.

Our revised guidance therefore specifies how to act on the output. Users can use sparse placement to prioritize candidate comparisons, then apply the follow-up measurement procedure when peer identity matters. The reported precision and coverage make the reliability rule’s observed performance explicit. Decisions requiring 90% precision should retain additional verification.


Changes in the revised manuscript

Section 5.4 (p. 8), with the follow-up protocol and results in Appendix W. We explain how the shortlist is used and report its performance after additional measurement.

“The Top-5 are candidates for follow-up. In a separate O3 replay, 24-item follow-up recovered 3.510 versus 3.542 peers for SoftMap and calibration (Appendix W).”

Abstract (p. 1). We report the accepted-placement outcomes in both cohorts and their relation to the registered precision target.

“Its acceptance rule resolved 17 of 21 accepted placements on the first cohort and 14 of 16 on the second. Neither reached the registered 90% precision.”

Discussion, “How to use the map” (p. 10). We give the follow-up procedure and identify its retrospective validation.

For closer peer identification, nominate ten references, rerank them using 24 follow-up items, and return five; cached reference responses leave 24 additional target responses. We evaluated this procedure retrospectively on 192 disjoint items.

The same Discussion paragraph reports the before/after comparison:

Before follow-up, the corresponding means were 3.302 and 3.344. The gains of 0.240 ([−0.135, 0.635]) and 0.167 ([−0.188, 0.573]) remain unresolved after multiplicity correction. These outcomes quantify retrieval under the tested procedure; they do not establish an improvement from additional measurement.

Thanks to the authors for their detailed response and more experiments. However, my principal concerns still stand, namely that SoftMap achieves an NDCG@5 of 0.8609, compared with 0.8611 for calibration-only and 0.8646 for nuisance-only. Moreover, the reported incremental gain of 0.00789 from itemwise residual information beyond calibration is not significant after multiplicity correction. Thus, while the new permutation control shows that SoftMap uses question correspondence, the results still do not demonstrate a reliable advantage over simpler aggregate-feature approaches. Also, exact Top-5 retrieval is not reliably validated under the benchmark-disjoint capability definition, for NDCG@5 - SoftMap improves over Soft-JS by only +0.0080 and for Recall@5 - SoftMap improves by only +0.0008. Therefore, I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.