- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CapAtlas evaluates sparse behavioral model mapping under unseen-lineage transfer. It distinguishes global ranking, local retrieval, reliability estimation, and fixed-precision acceptance. The reported improvements over response-space mapping are positive, but calibration-only features achieve comparable results, and the prospective precision criterion is not met.
Critical strengths
1.The frozen, unseen-lineage evaluation design reduces direct target leakage.
2.Separate evaluation of retrieval and reliability prevents overinterpreting a single mapping score.
Critical weaknesses
-
The main representation does not consistently outperform simple calibration features. Table 1 reports NDCG@5 of 0.8609 for SoftMap versus 0.8611 for calibration-only features. Table 2 similarly reports 0.8310 versus 0.8372 under the benchmark-disjoint capability construct. These results leave the additional value of itemwise behavioral geometry unclear, despite the greater complexity of the mapping pipeline.
-
The prospective reliability target is not achieved. The method reaches 80.95% precision at 42% coverage, below the registered 90% requirement. Both oracle-half evaluations also fall below the target. Although AUROC of 0.731 indicates useful risk ranking, the evidence does not establish a reliable fixed-precision acceptance policy.
Questions for authors
1.Which domains show statistically significant improvements from itemwise features beyond calibration-only representations?
2.What practical advantage justifies the additional complexity over calibration-only mapping?
3.Is the failed precision criterion mainly attributable to risk estimation or threshold calibration?
4 comments
Response to Reviewer ypWb (2/4): W1/Q2 - Manuscript changes
W1/Q2. Methodological contribution and practical value (continued)
Changes in the revised manuscript
Section 3.2, p. 4 — information retained by the two representations.
Calibration pools responses into mean entropy, maximum probability, and four option probabilities. SoftMap retains which response distribution belongs to which question. Models with identical pooled summaries can therefore have distinct itemwise representations.
Location 1: Appendix U.1, p. 29 — controlled test of question correspondence.
Independently permuting the 16 item blocks within each model preserves mean entropy, maximum probability, and all four option means: each is a symmetric sum over items.
Location 2: Section 5.3, p. 8 — measured contribution of item alignment.
Across 100 fixed permutations with retraining, aligned SoftMap gains 0.05144 NDCG ([0.03175, 0.07338], Holm p=.000020) and 0.12234 Recall ([0.07484, 0.17499], Holm p=.00032). A common permutation of all models leaves predictions unchanged to numerical precision.
Location 3: Introduction, p. 1 — practical implementation and intended output.
For a new checkpoint we recommend starting with calibration: average its entropy, maximum probability and four answer-option probabilities over the same 16 probes. Fit the map from these six features to measured reference profiles on the atlas alone. This yields a coarse ordering and a list of candidates to measure.
The surrounding discussion in Section 5.3 and Appendix U.1 distinguishes the alignment result from incremental superiority over calibration. Section 5.4 (p. 8) explains the role of the Top-5 as follow-up candidates. Appendices V–W (pp. 32–34) report the frozen new-program comparison and the retrospective follow-up evaluation, respectively.
Response to Reviewer ypWb (3/4): Q1
Q1. Domain-specific benefits beyond calibration
Author response
Thank you for asking us to identify where itemwise information provides benefits beyond calibration. No domain-level improvement survives multiplicity correction in the current evaluations. We report the positive estimates as exploratory findings and distinguish them from the controlled evidence that SoftMap uses question correspondence.
In the original five-domain comparison (Appendix T), SoftMap shows positive NDCG@5 differences over calibration in biomedical tasks (+0.03998) and general reasoning (+0.01989). Their unadjusted p-values are 0.04543 and 0.04721, respectively, but both become 0.22715 after Holm correction across the five domains. Thus, these estimates identify possible areas of benefit without establishing a domain-specific advantage.
We also tested incremental itemwise information more directly by adding a reference-fitted itemwise residual predictor to calibration (C+I). In the supplementary fixed-panel analysis (Appendix U.1), this addition improves mathematical-domain NDCG@5 by 0.0190, but the contrast does not survive correction (unadjusted p=0.0196; Holm-adjusted p=0.3101). None of the 18 aggregate/domain conditional NDCG comparisons in that analysis survives its correction. The original five-domain tests and the supplementary 18-comparison analysis are reported as separate correction families.
These results clarify the distinction between the methodological contribution and a domain-specific recommendation. The alignment control discussed in our W1/Q2 response establishes that question correspondence contributes to SoftMap’s predictions. Identifying a domain in which SoftMap should be preferred to calibration requires a direct, corrected comparison against calibration; the current domain results do not establish such a preference. We retain the complete domain results, label the positive differences as exploratory, and do not use them to prescribe domain-specific estimator selection.
Changes in the revised manuscript
Location 1: Appendix T, p. 28 — conclusion of the original five-domain comparison.
No domain passes Holm correction.
The table reports all five domains, including their estimated differences, confidence intervals, unadjusted p-values, and Holm-adjusted p-values.
Location 2: Appendix U.1, Table 21, p. 29 — correction scope for the supplementary comparisons.
The first 18 NDCG rows share one correction family; the final two share another.
This table includes the direct SoftMap–calibration comparisons and the incremental C+I–calibration comparisons, preventing an isolated unadjusted result from being interpreted as a confirmed domain-level benefit.
Location 3: Appendix U.1, p. 29 — interpretation of the alignment experiment.
These controls show use of question correspondence, not incremental superiority over C or N.
Here, C denotes calibration-only and N denotes the metadata-plus-calibration nuisance model. Section 5.3 also distinguishes exploratory domain effects from the contribution of item alignment.
Response to Reviewer ypWb (4/4): W2/Q3
W2/Q3. Risk estimation versus threshold calibration
Author response
Thank you for asking us to distinguish these possible explanations. We examined discrimination, probability calibration, and threshold transfer separately. The current evidence does not support attributing the failed precision target solely to threshold calibration, nor does it identify the relative contribution of each source of error.
The original threshold was chosen using an empirical 80% development-prefix criterion, whereas the prospective evaluation required 90% precision. It was therefore not a development-validated 90% policy. The frozen rule resolved 17/21 accepted placements in the first cohort and 14/16 on the new programs. Its AUROC was 0.731 and 0.564, respectively, indicating weaker observed discrimination in the new cohort. Development-fitted sigmoid recalibration changed the first external Brier score only slightly, from 0.21490 to 0.21306; this monotonic recalibration does not change the score ordering. Three development-only 90% alternatives abstained on the external evaluations, leaving precision undefined.
In a retrospective threshold scan, the widest first-cohort prefix reaching 90% observed precision resolved 14/15. Applying that cutoff unchanged to the new cohort resolved 4/4 at 12.5% coverage (one-sided 95% exact lower bound 47.3%, assuming independent outcomes). Selecting a cutoff with access to the new labels gave 10/11 at 34.4% coverage. A stricter threshold can reduce coverage sharply; the data still leave the relative contributions of score quality and threshold transfer unresolved (Appendices U–V).
The post-hoc scan evaluates all distinct cutoffs and keeps tied scores together. The widest qualifying prefix on the first cohort has cutoff 0.70365138 and covers 15/50 (30%). On the new cohort the four selected targets belonged to four programs. The threshold chosen with access to the new labels was 0.65217850; its 10/11 is an in-sample descriptive upper envelope for observed coverage at the requested precision on these scores. Both cohorts had already been analyzed before designing this diagnostic. We retained the original frozen-rule result and do not propose either scanned cutoff as a validated policy.
Changes in the revised manuscript
Locations: Section 5.5; Appendix U, Table 24 (p. 31); Appendix V (pp. 32–33).
Revised text (Appendix V, p. 33):
Both cohorts were already analyzed when we designed this diagnostic. It quantifies the precision–coverage tradeoff and leaves the registered result and the source of the transfer error unresolved.
Supporting passage (Appendix V, p. 32):
We retrospectively evaluated every distinct cutoff of the original frozen SoftMap risk score, accepting all ties. On the first 50 targets the widest score prefix with observed precision of at least 90% contained 15 targets, 14 resolved (30% coverage, cutoff 0.70365138, applied at full precision). Applying that same cutoff to the 32 completed new targets accepted four from four programs; all four resolved (12.5% coverage). Even treating those four outcomes as independent Bernoulli trials, their one-sided 95% exact lower bound is 47.3%. Choosing afresh with access to the new labels yields 10/11 at a cutoff of 0.65217850 (34.4% coverage).
Response to Reviewer ypWb (1/4): W1/Q2 - Author response
W1/Q2. Methodological contribution and practical value
Author response
Thank you for asking us to clarify both the methodological contribution of the itemwise representation and its practical value relative to calibration-only mapping.
The methodological distinction concerns what information is retained. Calibration pools responses across probes into mean entropy, maximum probability, and four answer-option probabilities. SoftMap instead retains the association between each question and its response distribution, and learns a low-rank mapping from these itemwise patterns to measured reference capabilities. Consequently, two models can have identical calibration summaries while exhibiting different question-specific response patterns; SoftMap can represent this distinction, whereas the six-summary representation cannot. Both methods use reference capability supervision. The additional modeling capacity of SoftMap is therefore its ability to use question-specific behavioral structure beyond aggregate confidence and option preferences.
We directly tested whether this structure contributes to SoftMap’s predictions. Independently permuting the 16 item blocks within each model preserves all six calibration summaries but disrupts cross-model question correspondence. Across 100 fixed permutations with refitting, aligned SoftMap exceeds this control by 0.05144 NDCG@5 (95% CI [0.03175, 0.07338]; Holm-adjusted p=0.000020) and 0.12234 Recall@5 ([0.07484, 0.17499]; Holm-adjusted p=0.00032). Applying a common permutation to all models leaves predictions unchanged to numerical precision. These controls show that question correspondence, rather than the arbitrary ordering of questions, makes a measurable contribution within SoftMap.
This finding addresses a different question from whether SoftMap outperforms calibration. We agree that the latter advantage is not established consistently: calibration matches or exceeds SoftMap’s aggregate NDCG@5 under both O1 and O2. Adding reference-fitted itemwise residual prediction to calibration yields a gain of 0.00789 NDCG@5 on the original panel, but this contrast does not survive multiplicity correction (p=0.310). We therefore distinguish evidence that the representation uses question-specific information from evidence that retaining this information improves prediction over a strong pooled baseline.
The practical contribution of CapAtlas is to connect a new checkpoint’s sparse responses to models with measured capabilities, producing candidates for further comparison. We evaluated this use beyond the original aggregate scores. On 32 completed targets from 24 new training programs, the frozen estimators used the same 16 probes and achieved NDCG@5 of 0.9072 for SoftMap and 0.9036 for calibration; the paired difference was 0.0036 (95% CI [−0.021, 0.024]). In a retrospective follow-up evaluation on these targets, each estimator nominated ten references, used 24 additional items to rerank them, and returned five. Evaluation on 192 disjoint items yielded program-weighted means of 3.510 true peers for SoftMap and 3.542 for calibration, with a paired difference of −0.031 [−0.208, 0.156]. The O3 calibration comparator additionally includes the mean top-two probability margin. Before follow-up, the corresponding means were 3.344 for SoftMap and 3.302 for calibration. The within-policy gains were +0.167 (95% CI [−0.188, 0.573]) and +0.240 [−0.135, 0.635], respectively; neither survived multiplicity correction. The replay therefore quantifies peer recovery under the tested procedure without establishing a benefit from the additional measurement.
These results provide a concrete implementation recommendation: use calibration for coarse placement and measure the candidates when exact peer identity matters. SoftMap provides a way to represent and test question-specific behavioral information, and the alignment experiment establishes its role within the method. The direct baseline and follow-up comparisons determine whether that additional capacity is worth using in the evaluated setting. We have revised the manuscript to make this methodological distinction explicit and to align the recommended implementation with the observed practical results.