- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how many intent descriptions should be shown to a frozen language model ("reader") for multi-intent detection in a new domain, where only the label schema is available and no target examples may be used. The authors factorize exact-set accuracy into gold coverage (query-dependent) and reader preservation (reader-dependent), and show that the best fixed exposure depth varies with both reader and schema. CAPE estimates the two terms separately: a source-trained selector predicts how deep the ranked list must go to cover the gold set, and a label-free calibration — self-tests built from the schema itself, queried at increasing depths — estimates how many additional distractor labels the deployed reader tolerates. A per-query update from prediction agreement under two orderings of the same pool refines this estimate, and a coverage–retention score selects the final exposure. Across 72 local settings (nine readers, four schemas, two suites), CAPE attains the best macro Exact (44.91 vs. 40.01 for fixed top-20) and improves by +0.86 points over a reader-calibrated threshold policy under matched budgets (Table 2). Gains are consistent on the two larger schemas; results on smaller schemas are reader-dependent, and on both hosted APIs CAPE trails Full exposure overall (Table 4). The paper is notable for self-diagnosis, auditing and reporting its own coverage overprediction (Table 28).
Critical strengths
S1 — A well-posed problem and a genuinely new measurement capability. The coverage–preservation factorization is clean, and the label-free reader calibration is, to my knowledge, the first behavioral proxy for the reader-side term that requires no target labels, model internals, or per-reader training: the reader's own answers to schema-derived self-tests reveal how many distractors it tolerates. The proxy predicts held-out readers' responses better than an empirical reference (rank correlation 0.62 → 0.76; Tables 22–23), and the full policy beats a reader-calibrated threshold control under matched budgets while exposing fewer labels (Table 2).
S2 — Exceptional experimental hygiene and self-diagnosis. Budgets are matched across all comparisons; six of nine readers (including a cross-family model family) and both hosted APIs are held out from calibration fitting; seed, precision, and naturalization controls are reported. Most tellingly, failures are quantified rather than hidden: the paper itself reports where its coverage estimates are over-optimistic, where it loses to Full exposure, and how its gains and losses decompose (Tables 4, 6, 11, 28).
Critical weaknesses
W1 — The gain over simpler calibrated policies is small relative to the machinery required. Under matched budgets, CAPE improves Exact by +0.86 over a coverage-threshold policy, and by only +0.49 over its own simplified variant, whose held-out advantage over the threshold policy is not statistically resolved (Tables 2, 17). The full procedure is not cheaper either: it spends more calls and tokens per query than Full exposure at decision time (Table 5), and its reported intervals condition on a fixed calibration, so they do not speak to stability across recalibration or new readers (App. M.4). Whether Fisher-designed probes, EVI allocation, and the per-query Bayes update justify their complexity is the central question for adoption.
W2 — No reliable way to decide when Full exposure should be preferred. The authors themselves document that benefits are uneven — consistent gains on the two larger schemas (Table 27), regressions on ATIS where the coverage estimate is over-optimistic (88.9% predicted vs. 67.2% observed; Table 28), and both hosted APIs trailing Full (Table 4) — and this transparency is to their credit. What is missing is the decision rule that this evidence calls for. Omitting a gold label is an unrecoverable error, so a deployment needs a label-free way to know when to keep the full schema; neither the coverage floor of App. P (chosen after inspecting ATIS results) nor the arbitration variant (which costs −0.93 Exact overall; Table 31) provides one. A principled boundary — even a simple diagnostic — would substantially strengthen the case for deployment.
Questions for authors
Q1 Has the root cause of the ATIS coverage overprediction been analyzed further (e.g., encoder ranking quality or transfer of the rank features across schemas)? Relatedly, is there a label-free diagnostic that could flag unsafe coverage estimates before deployment, given that the 0.90 coverage floor in Appendix P was selected after inspecting ATIS results?
Q2 Calibration-call allocation currently uses the complete unlabeled target test batch (Appendix D.2), and Section 7 notes that online use requires a representative unlabeled buffer. Is there any experiment (or bound) on how allocation and final performance degrade with smaller or shifted unlabeled buffers in the streaming setting?
Q3 Given the component-level deltas (Tables 2 and 17), can you state when a practitioner should prefer the simplified CAPE basic, the threshold policy, or simply Full exposure, over the full procedure — for example, a rule based on schema size and estimated coverage quality?
4 comments
2/2
Q1: What further evidence explains ATIS coverage overprediction?
As your review notes, Table 28 already reports the 88.9% predicted versus 67.2% observed coverage. We have retained that result and now added the further diagnosis in Appendix Q.1, Tables 32–35:
- Table 32: complete-gold coverage on ATIS is 68.4% at and 88.7% at . Some queries require substantially deeper exposure even in this 18-intent schema.
- Table 33: ATIS's mean overprediction gap is +21.7 pp, compared with +2.5/+5.6/+3.3 pp for SNIPS/Banking77/CLINC150.
- Table 34: relative to source validation, ATIS's top-two similarity gap rises from 0.082 to 0.114, while the gap after the last gold intent falls from 0.054 to 0.011. Early separation can coexist with weak contrast at the actual gold boundary.
- Table 35: gold omission accounts for 76.5% of ATIS Full-correct-to-CAPE-wrong transitions, increasing to 86.8% for Qwen3-4B/8B/14B/32B.
These observations support a coverage-transfer explanation; they do not identify a unique causal feature or show that ATIS has the worst retrieval across schemas. The gold-boundary statistic is retrospective and is not an input to the Full switch. Section 6 now makes that distinction explicit.
Q2: How does performance change with smaller or shifted allocation buffers?
Access to the complete unlabeled target test batch was already explicitly specified in Section 5 and Appendix D.2. Label-free therefore means no target annotations, not an absence of target utterances. The additional evidence needed concerns performance with smaller or shifted buffers.
We have now added that experiment in Appendix Q.4, Tables 39–40, using Qwen3-8B/Llama-3.2-3B on all four domains and both suites. A fixed 400-query evaluation set is excluded from the allocation reservoir. Relative to the disjoint full-reservoir reference, the changes over three buffer seeds are:
| Buffer | Change in Exact, pp (mean ± SD) |
|---|---|
| Random 32 | |
| Random 128 | |
| Random 512 | |
| Shifted 128 |
The shifted buffer samples the top quartile of predicted . Table 40 verifies that its mean predicted rises from 0.136 to 0.247, without selecting examples using gold labels. Final mean exposure increases by 0.22. These results directly test small buffers and a specified allocation-distribution shift; Section 7 limits the conclusion accordingly rather than claiming robustness to arbitrary temporal drift.
Q3: When should a practitioner choose Full, threshold, basic, or CAPE?
We have now made the policy guidance explicit in Section 6, combining the comparisons already present in Tables 2, 17, and 27 with the newly measured runtimes in Tables 41–42:
- Full: when a one-call, low-latency budget is the priority.
- Threshold: when a simple calibrated coverage cutoff is preferred.
- Basic: when capacity estimation is useful but Fisher/EVI selection and query updating are undesirable. Threshold and basic both use three-view aggregation.
- CAPE: when prioritizing accuracy among the three-call calibrated policies, with the strongest evidence on the tested larger schemas (Tables 2 and 27); the input-only diagnostic above provides an optional source-calibrated Full fallback.
The reader-by-domain breakdown was already included in Table 27 and identifies where the accuracy evidence is strongest: all nine readers have positive CAPE differences over both matched controls on Banking77 and CLINC150, while ATIS is less favorable. This is empirical guidance, not a claim that schema size or reader size alone identifies the best policy.
We would appreciate reconsideration in light of the new pre-query Full-selection rule, explicit ATIS diagnosis, independent recalibration results, disjoint-buffer evaluation, and measured execution trade-offs.
Thank you for the detailed response and the additional experiments. The recalibration repeats, disjoint-buffer evaluation, and measured runtime meaningfully address several of my original concerns. I also appreciate the clear reporting of limitations and negative results.
My remaining concern is primarily about the incremental contribution of the full framework. The reader-calibrated threshold control already achieves 44.05 Exact, compared with 44.42 for basic and 44.91 for full CAPE. I recognize that the threshold control is constructed within this work and shares important components with CAPE. Its strong performance supports the value of schema-derived calibration, but also leaves me uncertain how much additional methodological value is established by explicit capacity modeling, Fisher/EVI selection, and query-level updating.
The runtime measurements helpfully show that the full procedure adds little online overhead relative to the matched three-call controls. I therefore consider the efficiency concern substantially reduced. However, the modest accuracy gains still leave a question about whether the additional mechanisms provide sufficient benefit over a simpler calibrated policy.
It would help me to understand the authors’ view of the central contribution: is it principally schema-derived calibration for exposure selection, or the explicit measurement and adaptation of reader capacity? In particular, what is the strongest evidence that the latter provides useful predictive or decision-making insight beyond selecting an exposure threshold on the same synthetic probes?
I am retaining my current rating for now, while remaining open to further discussion on this point. The additional evidence has strengthened my confidence in the empirical results; my remaining reservation concerns the significance of the methodological increment.
Thank you for clarifying the remaining concern. We appreciate your assessment that the additional experiments address the empirical and efficiency questions. Your distinction between reliable results and the significance of the methodological contribution is helpful.
Our central contribution is the behavioral measurement of reader-specific exposure tolerance from schema-derived self-tests, and its use for predicting reader responses and selecting exposure. Schema-derived calibration supplies the observations; the capacity distribution represents what those observations imply about tolerance to additional labels. Fisher/EVI selection and query-level updating are refinements of this measurement-and-decision framework. We agree that their smaller incremental gains should be distinguished from the broader contribution.
Both CAPE and the calibrated threshold policy choose query-dependent depths. The difference is what calibration estimates. Thresholding selects a reader–schema-specific coverage cutoff. CAPE estimates a retention curve describing how the reader preserves correctness as additional labels are exposed beyond gold coverage.
For each candidate exposure depth, CAPE considers the possible ranks at which all correct intents become available. It weights the estimated retention at each corresponding excess depth by the predicted probability of that gold-cover rank, summing over ranks included in the candidate pool. This produces a decision surrogate that accounts for both including the correct intents and preserving correctness as exposure grows.
The substantive question is whether this estimated response structure has predictive and decision value. The strongest evidence is already included in the following experiments, which we should have connected more directly to this question.
1. Capacity predicts real-query exposure responses before query-level adaptation.
Figure 4 and Tables 22–23 evaluate six readers excluded from source response-model fitting. On real queries, conditional on an exact anchor response, CAPE improves reader-rank correlation from 0.620 to 0.762 and reduces retention-curve MAE from 0.231 to 0.188 relative to the empirical stable-depth reference. These predictions use the pre-query capacity posterior, before observing the real query's order-agreement signal. Thus, the estimated capacity captures variation in how unseen readers respond to increased exposure, beyond reporting which synthetic threshold performed best.
The separate decision audit in Table 22, using all sampled queries, reduces regret relative to the best observed query-grid action from 3.95 to 1.95 pp and raises Exact from 39.20 to 41.20 against that empirical reference. This tests whether the measured response structure supports useful exposure decisions.
2. Matching the capacity estimate to the reader matters.
Table 3 already includes a posterior-shuffling intervention that fixes queries, rankings, and tested readers while exchanging reader-posterior assignments within each target–suite. Exact falls from 44.91 to 43.40, even though gold coverage increases from 71.99% to 72.97%. This supports the decision value of reader-specific information in the posterior. It does not, by itself, isolate every component or replace the direct threshold comparison.
3. The direct threshold comparison also shows calibration-budget value.
The comparisons above use diagnostic references. Figure 3 directly compares CAPE with the calibrated threshold policy using the shared synthetic candidate bank and three calls per real query. CAPE with a 32-call calibration budget exceeds thresholding with 64 calls by +0.48 pp [95% CI: +0.18, +0.79] overall and +0.59 pp [+0.15, +1.03] on held-out readers. This supports the usefulness of the complete capacity-based procedure under a smaller synthetic-classification budget; it does not attribute the difference to Fisher/EVI alone.
We agree that the additional mechanisms provide refinements rather than uniformly large gains. Table 17 quantifies the full-versus-basic increment, while its midpoint-capacity control more specifically tests retaining interval uncertainty: full CAPE gains +0.46 pp [+0.17, +0.74] over midpoint capacity overall. Our claim is therefore that capacity is a measurable, predictively useful representation for exposure decisions, with additional modest benefits from uncertainty handling and adaptation. The strong threshold result establishes a useful simpler alternative within the same framework.
We have updated the Introduction to clarify the central contribution and the roles of the additional components, and Section 6 to explain the complementary predictive and policy evidence. Changes are marked in blue. We hope these clarifications address your remaining concern and would appreciate your reassessment in light of this evidence.
1/2
Thank you for identifying the deployment questions requiring further evidence. We build on the matched-policy and failure analyses already included in the paper and have now added a Full-selection diagnostic, independent recalibration repeats, small/shifted-buffer tests, and measured runtime. Below we address W1–W2 and Q1–Q3 directly.
W1: Is the improvement over simpler calibrated policies worth the complexity, and are the results stable across recalibration?
We agree that the incremental gain over basic is modest. As you noted, Tables 2 and 17 already provide this simplified-policy comparison: CAPE exceeds basic by +0.49 pp [0.22, 0.78] overall and +0.52 pp [0.14, 0.95] on held-out readers. Your observation about basic versus thresholding is correct: Table 17, Panel D, reports a held-out difference of +0.38 pp with CI [−0.04, +0.81]. That unresolved comparison is distinct from the positive interval for full CAPE versus basic. Section 6 now separates the larger benefit of reader calibration from the smaller incremental benefit of the full procedure.
The conditional scope of the query-bootstrap CIs is already stated in Appendix M.4, and we agree that those intervals do not measure recalibration variability. We have now added a separate recalibration test in Appendix Q.3, Table 38: three independently regenerated probe banks, rerunning each policy's calibration on Qwen3-8B and Llama-3.2-3B for ATIS/CLINC150 under Blend/Mix. Source components and the 400-query evaluation subset per setting are fixed. Paired gains are +0.78±0.05 pp over threshold and +0.67±0.05 pp over basic, positive in every repeat. These are means ± sample SDs across banks; we do not reinterpret the conditional CIs as covering all training or reader variability.
We have also added measured costs in Appendix Q.5, Tables 41–44. On RTX 5090/BF16 with sequential calls, CAPE adds 9.3 ms (1.93%) / 4.7 ms (1.70%) over basic for the two readers, alongside +0.75/+0.57 pp Exact in this audit. Its total latency is 2.29/2.19 times Full. This quantifies the incremental execution cost of the full procedure while preserving the distinction from the cheaper one-call policy.
W2: Is there a label-free rule for preferring Full?
To address the missing pre-query decision rule, we have now added an input-only switch in Section 6 and Appendix Q.2, Tables 36–37. This is a new diagnostic, separate from Appendix P's target-informed floor and verifier-based arbitration.
The diagnostic uses four features: top-1 similarity, top-two similarity gap, predicted mean cover rank divided by schema size, and normalized entropy of the predicted rank distribution. One source split estimates feature means/scales; a disjoint source split fixes the 95th percentile of the RMS standardized distance. A query above this cutoff uses Full; otherwise it uses native CAPE. Both source splits exclude the target domain. The switch is decided before query-level reader calls, without target gold labels or reader predictions.
Table 36 reports:
Random switching at the same per-setting rates gives expected Exact 44.46 (−0.45 pp). Thus the diagnostic adds selection value beyond using Full more often. Table 37 further shows higher native gold-omission rates among flagged queries in every domain; on ATIS these are 68.0% versus 25.4% for retained queries. Gold labels are used only to evaluate these outcomes.
Table 36 also reports small negative point changes on the other three domains (−0.14 to −0.28 pp, with CIs crossing zero). The switch therefore provides an empirically useful fallback condition, not a guarantee of harmless or optimal selection.