acceptodds
Under review as a conference paper at ICLR 2027

HOW MUCH SCHEMA CAN A READER USE? CAPACITY-ADAPTIVE TEST-TIME EXPOSURE FOR OUT-OF-DOMAIN MULTI-INTENT DETECTION

Abstract

When a frozen language model, or reader, predicts intents from label descriptions, showing more labels can help it find the correct ones but also introduce distractions. How many labels to show depends on both the query and the reader. We introduce Capacity-Adaptive Pool Exposure (CAPE) for multi-intent detection in new domains, without labeled target examples or access to reader internals. A source-trained selector estimates whether a candidate pool contains all required labels. Tests built from target label descriptions measure how well each reader handles additional labels, and prediction agreement under reordered descriptions refines this estimate for each query. Across 72 settings spanning nine local readers and four domains, CAPE improves macro exact-set accuracy (Exact) by 12.2% relative to fixed top-20 exposure, displaying 11.3 labels on average for the final prediction. With matched calibration and per-query call budgets, it improves Exact by 2.0% overall and 2.2% on held-out readers relative to reader-calibrated coverage thresholding. These results show that measuring reader responses to additional labels improves exposure decisions beyond estimating label coverage alone.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Test-time adaptation, Black-box language models, Adaptive schema exposure, Multi-intent detection, Behavioral calibration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_N9qaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how many intent descriptions should be shown to a frozen language model ("reader") for multi-intent detection in a new domain, where only the label schema is available and no target examples may be used. The authors factorize exact-set accuracy into gold coverage (query-dependent) and reader preservation (reader-dependent), and show that the best fixed exposure depth varies with both reader and schema. CAPE estimates the two terms separately: a source-trained selector predicts how deep the ranked list must go to cover the gold set, and a label-free calibration — self-tests built from the schema itself, queried at increasing depths — estimates how many additional distractor labels the deployed reader tolerates. A per-query update from prediction agreement under two orderings of the same pool refines this estimate, and a coverage–retention score selects the final exposure. Across 72 local settings (nine readers, four schemas, two suites), CAPE attains the best macro Exact (44.91 vs. 40.01 for fixed top-20) and improves by +0.86 points over a reader-calibrated threshold policy under matched budgets (Table 2). Gains are consistent on the two larger schemas; results on smaller schemas are reader-dependent, and on both hosted APIs CAPE trails Full exposure overall (Table 4). The paper is notable for self-diagnosis, auditing and reporting its own coverage overprediction (Table 28).

Critical strengths

S1 — A well-posed problem and a genuinely new measurement capability. The coverage–preservation factorization is clean, and the label-free reader calibration is, to my knowledge, the first behavioral proxy for the reader-side term that requires no target labels, model internals, or per-reader training: the reader's own answers to schema-derived self-tests reveal how many distractors it tolerates. The proxy predicts held-out readers' responses better than an empirical reference (rank correlation 0.62 → 0.76; Tables 22–23), and the full policy beats a reader-calibrated threshold control under matched budgets while exposing fewer labels (Table 2).

S2 — Exceptional experimental hygiene and self-diagnosis. Budgets are matched across all comparisons; six of nine readers (including a cross-family model family) and both hosted APIs are held out from calibration fitting; seed, precision, and naturalization controls are reported. Most tellingly, failures are quantified rather than hidden: the paper itself reports where its coverage estimates are over-optimistic, where it loses to Full exposure, and how its gains and losses decompose (Tables 4, 6, 11, 28).

Critical weaknesses

W1 — The gain over simpler calibrated policies is small relative to the machinery required. Under matched budgets, CAPE improves Exact by +0.86 over a coverage-threshold policy, and by only +0.49 over its own simplified variant, whose held-out advantage over the threshold policy is not statistically resolved (Tables 2, 17). The full procedure is not cheaper either: it spends more calls and tokens per query than Full exposure at decision time (Table 5), and its reported intervals condition on a fixed calibration, so they do not speak to stability across recalibration or new readers (App. M.4). Whether Fisher-designed probes, EVI allocation, and the per-query Bayes update justify their complexity is the central question for adoption.

W2 — No reliable way to decide when Full exposure should be preferred. The authors themselves document that benefits are uneven — consistent gains on the two larger schemas (Table 27), regressions on ATIS where the coverage estimate is over-optimistic (88.9% predicted vs. 67.2% observed; Table 28), and both hosted APIs trailing Full (Table 4) — and this transparency is to their credit. What is missing is the decision rule that this evidence calls for. Omitting a gold label is an unrecoverable error, so a deployment needs a label-free way to know when to keep the full schema; neither the coverage floor of App. P (chosen after inspecting ATIS results) nor the arbitration variant (which costs −0.93 Exact overall; Table 31) provides one. A principled boundary — even a simple diagnostic — would substantially strengthen the case for deployment.

Questions for authors

Q1 Has the root cause of the ATIS coverage overprediction been analyzed further (e.g., encoder ranking quality or transfer of the rank features across schemas)? Relatedly, is there a label-free diagnostic that could flag unsafe coverage estimates before deployment, given that the 0.90 coverage floor in Appendix P was selected after inspecting ATIS results?

Q2 Calibration-call allocation currently uses the complete unlabeled target test batch (Appendix D.2), and Section 7 notes that online use requires a representative unlabeled buffer. Is there any experiment (or bound) on how allocation and final performance degrade with smaller or shifted unlabeled buffers in the streaming setting?

Q3 Given the component-level deltas (Tables 2 and 17), can you state when a practitioner should prefer the simplified CAPE basic, the threshold policy, or simply Full exposure, over the full procedure — for example, a rule based on schema size and estimated coverage quality?

4 comments

Thank you for identifying the deployment questions requiring further evidence. We build on the matched-policy and failure analyses already included in the paper and have now added a Full-selection diagnostic, independent recalibration repeats, small/shifted-buffer tests, and measured runtime. Below we address W1–W2 and Q1–Q3 directly.

W1: Is the improvement over simpler calibrated policies worth the complexity, and are the results stable across recalibration?

We agree that the incremental gain over basic is modest. As you noted, Tables 2 and 17 already provide this simplified-policy comparison: CAPE exceeds basic by +0.49 pp [0.22, 0.78] overall and +0.52 pp [0.14, 0.95] on held-out readers. Your observation about basic versus thresholding is correct: Table 17, Panel D, reports a held-out difference of +0.38 pp with CI [−0.04, +0.81]. That unresolved comparison is distinct from the positive interval for full CAPE versus basic. Section 6 now separates the larger benefit of reader calibration from the smaller incremental benefit of the full procedure.

The conditional scope of the query-bootstrap CIs is already stated in Appendix M.4, and we agree that those intervals do not measure recalibration variability. We have now added a separate recalibration test in Appendix Q.3, Table 38: three independently regenerated probe banks, rerunning each policy's calibration on Qwen3-8B and Llama-3.2-3B for ATIS/CLINC150 under Blend/Mix. Source components and the 400-query evaluation subset per setting are fixed. Paired gains are +0.78±0.05+0.78 \pm 0.05 pp over threshold and +0.67±0.05+0.67 \pm 0.05 pp over basic, positive in every repeat. These are means ± sample SDs across banks; we do not reinterpret the conditional CIs as covering all training or reader variability.

We have also added measured costs in Appendix Q.5, Tables 41–44. On RTX 5090/BF16 with sequential calls, CAPE adds 9.3 ms (1.93%) / 4.7 ms (1.70%) over basic for the two readers, alongside +0.75/+0.57 pp Exact in this audit. Its total latency is 2.29/2.19 times Full. This quantifies the incremental execution cost of the full procedure while preserving the distinction from the cheaper one-call policy.

W2: Is there a label-free rule for preferring Full?

To address the missing pre-query decision rule, we have now added an input-only switch in Section 6 and Appendix Q.2, Tables 36–37. This is a new diagnostic, separate from Appendix P's target-informed floor and verifier-based arbitration.

The diagnostic uses four features: top-1 similarity, top-two similarity gap, predicted mean cover rank divided by schema size, and normalized entropy of the predicted rank distribution. One source split estimates feature means/scales; a disjoint source split fixes the 95th percentile of the RMS standardized distance. A query above this cutoff uses Full; otherwise it uses native CAPE. Both source splits exclude the target domain. The switch is decided before query-level reader calls, without target gold labels or reader predictions.

Table 36 reports:

ScopeCAPE ExactWith switchChange [95% CI], ppSwitch rate
ATIS37.4638.88+1.42 [+0.94, +1.91]17.4%
All 72 local settings44.9145.12+0.21 [+0.04, +0.38]9.7%

Random switching at the same per-setting rates gives expected Exact 44.46 (−0.45 pp). Thus the diagnostic adds selection value beyond using Full more often. Table 37 further shows higher native gold-omission rates among flagged queries in every domain; on ATIS these are 68.0% versus 25.4% for retained queries. Gold labels are used only to evaluate these outcomes.

Table 36 also reports small negative point changes on the other three domains (−0.14 to −0.28 pp, with CIs crossing zero). The switch therefore provides an empirically useful fallback condition, not a guarantee of harmless or optimal selection.

Q1: What further evidence explains ATIS coverage overprediction?

As your review notes, Table 28 already reports the 88.9% predicted versus 67.2% observed coverage. We have retained that result and now added the further diagnosis in Appendix Q.1, Tables 32–35:

  • Table 32: complete-gold coverage on ATIS is 68.4% at K=5K=5 and 88.7% at K=10K=10. Some queries require substantially deeper exposure even in this 18-intent schema.
  • Table 33: ATIS's mean overprediction gap is +21.7 pp, compared with +2.5/+5.6/+3.3 pp for SNIPS/Banking77/CLINC150.
  • Table 34: relative to source validation, ATIS's top-two similarity gap rises from 0.082 to 0.114, while the gap after the last gold intent falls from 0.054 to 0.011. Early separation can coexist with weak contrast at the actual gold boundary.
  • Table 35: gold omission accounts for 76.5% of ATIS Full-correct-to-CAPE-wrong transitions, increasing to 86.8% for Qwen3-4B/8B/14B/32B.

These observations support a coverage-transfer explanation; they do not identify a unique causal feature or show that ATIS has the worst retrieval across schemas. The gold-boundary statistic is retrospective and is not an input to the Full switch. Section 6 now makes that distinction explicit.

Q2: How does performance change with smaller or shifted allocation buffers?

Access to the complete unlabeled target test batch was already explicitly specified in Section 5 and Appendix D.2. Label-free therefore means no target annotations, not an absence of target utterances. The additional evidence needed concerns performance with smaller or shifted buffers.

We have now added that experiment in Appendix Q.4, Tables 39–40, using Qwen3-8B/Llama-3.2-3B on all four domains and both suites. A fixed 400-query evaluation set is excluded from the allocation reservoir. Relative to the disjoint full-reservoir reference, the changes over three buffer seeds are:

BufferChange in Exact, pp (mean ± SD)
Random 32−0.19±0.09-0.19 \pm 0.09
Random 128−0.07±0.05-0.07 \pm 0.05
Random 512−0.02±0.02-0.02 \pm 0.02
Shifted 128−0.31±0.07-0.31 \pm 0.07

The shifted buffer samples the top quartile of predicted K0/NK_0/N. Table 40 verifies that its mean predicted K0/NK_0/N rises from 0.136 to 0.247, without selecting examples using gold labels. Final mean exposure increases by 0.22. These results directly test small buffers and a specified allocation-distribution shift; Section 7 limits the conclusion accordingly rather than claiming robustness to arbitrary temporal drift.

Q3: When should a practitioner choose Full, threshold, basic, or CAPE?

We have now made the policy guidance explicit in Section 6, combining the comparisons already present in Tables 2, 17, and 27 with the newly measured runtimes in Tables 41–42:

  • Full: when a one-call, low-latency budget is the priority.
  • Threshold: when a simple calibrated coverage cutoff is preferred.
  • Basic: when capacity estimation is useful but Fisher/EVI selection and query updating are undesirable. Threshold and basic both use three-view aggregation.
  • CAPE: when prioritizing accuracy among the three-call calibrated policies, with the strongest evidence on the tested larger schemas (Tables 2 and 27); the input-only diagnostic above provides an optional source-calibrated Full fallback.

The reader-by-domain breakdown was already included in Table 27 and identifies where the accuracy evidence is strongest: all nine readers have positive CAPE differences over both matched controls on Banking77 and CLINC150, while ATIS is less favorable. This is empirical guidance, not a claim that schema size or reader size alone identifies the best policy.

We would appreciate reconsideration in light of the new pre-query Full-selection rule, explicit ATIS diagnosis, independent recalibration results, disjoint-buffer evaluation, and measured execution trade-offs.

Thank you for the detailed response and the additional experiments. The recalibration repeats, disjoint-buffer evaluation, and measured runtime meaningfully address several of my original concerns. I also appreciate the clear reporting of limitations and negative results.

My remaining concern is primarily about the incremental contribution of the full framework. The reader-calibrated threshold control already achieves 44.05 Exact, compared with 44.42 for basic and 44.91 for full CAPE. I recognize that the threshold control is constructed within this work and shares important components with CAPE. Its strong performance supports the value of schema-derived calibration, but also leaves me uncertain how much additional methodological value is established by explicit capacity modeling, Fisher/EVI selection, and query-level updating.

The runtime measurements helpfully show that the full procedure adds little online overhead relative to the matched three-call controls. I therefore consider the efficiency concern substantially reduced. However, the modest accuracy gains still leave a question about whether the additional mechanisms provide sufficient benefit over a simpler calibrated policy.

It would help me to understand the authors’ view of the central contribution: is it principally schema-derived calibration for exposure selection, or the explicit measurement and adaptation of reader capacity? In particular, what is the strongest evidence that the latter provides useful predictive or decision-making insight beyond selecting an exposure threshold on the same synthetic probes?

I am retaining my current rating for now, while remaining open to further discussion on this point. The additional evidence has strengthened my confidence in the empirical results; my remaining reservation concerns the significance of the methodological increment.

Thank you for clarifying the remaining concern. We appreciate your assessment that the additional experiments address the empirical and efficiency questions. Your distinction between reliable results and the significance of the methodological contribution is helpful.

Our central contribution is the behavioral measurement of reader-specific exposure tolerance from schema-derived self-tests, and its use for predicting reader responses and selecting exposure. Schema-derived calibration supplies the observations; the capacity distribution represents what those observations imply about tolerance to additional labels. Fisher/EVI selection and query-level updating are refinements of this measurement-and-decision framework. We agree that their smaller incremental gains should be distinguished from the broader contribution.

Both CAPE and the calibrated threshold policy choose query-dependent depths. The difference is what calibration estimates. Thresholding selects a reader–schema-specific coverage cutoff. CAPE estimates a retention curve describing how the reader preserves correctness as additional labels are exposed beyond gold coverage.

For each candidate exposure depth, CAPE considers the possible ranks at which all correct intents become available. It weights the estimated retention at each corresponding excess depth by the predicted probability of that gold-cover rank, summing over ranks included in the candidate pool. This produces a decision surrogate that accounts for both including the correct intents and preserving correctness as exposure grows.

The substantive question is whether this estimated response structure has predictive and decision value. The strongest evidence is already included in the following experiments, which we should have connected more directly to this question.

1. Capacity predicts real-query exposure responses before query-level adaptation.

Figure 4 and Tables 22–23 evaluate six readers excluded from source response-model fitting. On real queries, conditional on an exact anchor response, CAPE improves reader-rank correlation from 0.620 to 0.762 and reduces retention-curve MAE from 0.231 to 0.188 relative to the empirical stable-depth reference. These predictions use the pre-query capacity posterior, before observing the real query's order-agreement signal. Thus, the estimated capacity captures variation in how unseen readers respond to increased exposure, beyond reporting which synthetic threshold performed best.

The separate decision audit in Table 22, using all sampled queries, reduces regret relative to the best observed query-grid action from 3.95 to 1.95 pp and raises Exact from 39.20 to 41.20 against that empirical reference. This tests whether the measured response structure supports useful exposure decisions.

2. Matching the capacity estimate to the reader matters.

Table 3 already includes a posterior-shuffling intervention that fixes queries, rankings, and tested readers while exchanging reader-posterior assignments within each target–suite. Exact falls from 44.91 to 43.40, even though gold coverage increases from 71.99% to 72.97%. This supports the decision value of reader-specific information in the posterior. It does not, by itself, isolate every component or replace the direct threshold comparison.

3. The direct threshold comparison also shows calibration-budget value.

The comparisons above use diagnostic references. Figure 3 directly compares CAPE with the calibrated threshold policy using the shared synthetic candidate bank and three calls per real query. CAPE with a 32-call calibration budget exceeds thresholding with 64 calls by +0.48 pp [95% CI: +0.18, +0.79] overall and +0.59 pp [+0.15, +1.03] on held-out readers. This supports the usefulness of the complete capacity-based procedure under a smaller synthetic-classification budget; it does not attribute the difference to Fisher/EVI alone.

We agree that the additional mechanisms provide refinements rather than uniformly large gains. Table 17 quantifies the full-versus-basic increment, while its midpoint-capacity control more specifically tests retaining interval uncertainty: full CAPE gains +0.46 pp [+0.17, +0.74] over midpoint capacity overall. Our claim is therefore that capacity is a measurable, predictively useful representation for exposure decisions, with additional modest benefits from uncertainty handling and adaptation. The strong threshold result establishes a useful simpler alternative within the same framework.

We have updated the Introduction to clarify the central contribution and the roles of the additional components, and Section 6 to explain the complementary predictive and policy evidence. Changes are marked in blue. We hope these clarifications address your remaining concern and would appreciate your reassessment in light of this evidence.

Reviewer_LMsGOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is about multi intent detection, specifically for new domains. Usually people use a frozen LLM (which the authors call as reader) and show a candidate list of intent descriptions. The reader is given some top K intent (based on some similarity based model), and then the reader selects one. The paper is about how K is to be decided. Showing too few --- might miss out the correct intent. (coverage failure) Showing too many --- might add noise to the selection for the reader (retention or preservation failure)

One of the observations is that the best K depends on the model and also on the dataset and does not simply grows with model size.

Critical strengths
  • introduced Capacity-Adaptive Pool Exposure (CAPE) for multi-intent detection in new domains, without labeled target examples or access to reader internals.
  • The lemma 3 showing the factorization
  • Held-out evaluation on unseen sizes and unseen family
Critical weaknesses
  • From Table 3: Most of the uplift seems to be coming from the reader calibration, any comments on that?
  • What is the practical case for the method if the fairest gain is under one point? -- Table 2 and fig 3
  • The difference between CAPE and Full doesnt seem that large specifically for Qwen, GPT, Gemma, (except for Llama). What would be the suggestion in terms of using Full vs CAPE? Also, is it the case that GPT and Gemma are stronger models and can tolerate nearly anything/everything?

Writing:

  • Fig 1C: There should be a better way to have the legends, which is also leading to some confusion in understanding the figure.
Questions for authors

IN addition to the ones in the weakness:

  • Fig 1B: Why such a gap between the numbers for Llama-3.1-1B and 8B
  • As per the abstract there is a ~2% improvement in Exact, how effective is it? or how impactful is it?
4 comments

Thank you for the questions about the source and practical value of the gains. Below we clarify what the paper's existing results establish and distinguish them from the recalibration, runtime, and output analyses we have now added. Manuscript changes are marked in blue.

1. Does most of the improvement come from reader calibration?

Yes. As you noted, Table 3 already quantifies this pattern: reader adaptation raises Exact from 36.98 to 43.35, and combining reader and query adaptation reaches 44.91. Measuring reader-specific tolerance is a central contribution. The shuffled-posterior control in the same table lowers Exact to 43.40 despite higher coverage, supporting the importance of matching the measured capacity to the reader.

The comparison isolating the smaller incremental benefit is also already included in Tables 2 and 17: CAPE gains +0.86 pp over thresholding and +0.49 pp over basic, which omits Fisher/EVI selection and query updating. Section 6 now states this distinction explicitly.

2. How meaningful is a sub-one-point gain, and what does the abstract's approximately 2% mean?

The approximately 2% is relative improvement: Table 2's 44.05 to 44.91 is +0.86 percentage points, or +1.95% relative. These underlying accuracy values are already reported in Table 2 and have not changed. We have revised the Abstract to state both units explicitly, including +0.90 pp/+2.20% for held-out readers.

Two relevant checks are already included in the paper. Table 27 shows positive differences over both threshold and basic for every reader on Banking77 and CLINC150; this is consistency of point estimates, not significance for every setting. Figure 3 shows that CAPE with a 32-call calibration budget exceeds thresholding with a 64-call budget by +0.48 pp overall. We have now added two further checks addressing repeatability and incremental runtime:

  • Appendix Q.3, Table 38: across three independently regenerated probe banks on Qwen3-8B/Llama-3.2-3B and ATIS/CLINC150 under Blend/Mix, paired gains are +0.78±0.05+0.78 \pm 0.05 pp over threshold and +0.67±0.05+0.67 \pm 0.05 pp over basic, positive for all three banks. These are means ± sample SDs.
  • Appendix Q.5, Tables 41–42: CAPE adds 9.3 ms/1.93% and 4.7 ms/1.70% over basic, respectively, with Exact gains of +0.75/+0.57 pp in that audit.

The practical claim is a reproducible accuracy increment over simpler calibrated policies with a measured execution trade-off, rather than a large gain in every deployment.

3. When should Full be preferred, and do stronger readers tolerate essentially any schema?

The cases where Full outperforms CAPE are already explicitly reported in Tables 1, 4, and 11–12. For the hosted readers in Table 4—GPT-4o-mini and Gemini-3.1-Flash-Lite—Full is better overall, whereas CAPE has small positive gains over Full on Banking77/CLINC150. Figures 1 and 4 show why model strength alone is insufficient: preferred exposure and measured exposure-response behavior vary by reader and schema.

Section 6 now gives the operational interpretation: Full suits a one-call, low-latency budget; threshold/basic provide simpler three-call alternatives; CAPE offers the measured additional accuracy for deployments accepting its additional components. The runtime audit shows that CAPE's total latency is approximately 2.2–2.3 times Full.

To turn this existing failure evidence into a pre-query decision rule, we have now added a source-calibrated, input-only Full switch in Appendix Q.2, Tables 36–37. It uses similarity and predicted-rank-distribution features, with a threshold fixed on source data before querying the reader. Across the 72 local settings, it switches 9.7% of queries and improves Exact from 44.91 to 45.12; ATIS improves from 37.46 to 38.88. This provides a concrete diagnostic for choosing Full without target labels. It is evaluated on the local panel and does not guarantee the optimal policy for every query.

4. What do the markers and bars in Figure 1(c) represent?

We have expanded the Figure 1 caption while retaining the plotted results. Circles and triangles show the best tested K/NK/N for individual Qwen and Llama readers; diamonds show the mean reader-blind Adaptive-k fraction; bars show its interquartile range across queries. Thus the bars describe query-level exposure variation, not variation across readers.

5. Why do Llama 1B and 8B differ so much in Figure 1(b)?

The 1B model is Llama-3.2-1B. Section 3 and the Figure 1 caption now emphasize that panel (b) reports the best tested depth, not accuracy or calibrated capacity. The corresponding accuracy values were already reported in Table 11: 1B's best tested depth on CLINC150 is Full, but its Exact is only 4.35%. Llama-3.1-8B reaches 34.52% at K=10K=10, falling to 25.96% at Full.

What we have now added is the output analysis in Appendix Q.7, Table 46. At K=20K=20, 1B predicts 6.822 labels on average despite the prompt's maximum of three; raw responses include all 20 candidate IDs. There is no forced top-three truncation. At Full, output-error rates are 51.049% for 1B versus 0.147% for 8B. Over-three predictions are treated separately from output errors.

The gap in selected depths therefore occurs in very different reliability regimes: a large best-tested KK for 1B is not evidence that it reliably uses all 147 labels. These diagnostics clarify that interpretation without attributing the precise Full-versus-truncated ordering to a single proven cause.

We hope these clarifications make the existing calibration, matched-policy, and Figure 1 evidence easier to assess, while the added analyses answer the remaining stability and deployment questions. We would appreciate reconsideration in light of this combined evidence.

Thanks for the detailed update. I am increasing my score.

Thank you for taking the time to consider our responses and the additional evidence. We appreciate your constructive feedback and reassessment.

Reviewer_h1iPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

When a frozen LLM ("reader") picks intents from label descriptions, showing more labels has two opposing effects: it can help the reader find the correct ones, but it can also introduce distractions. The authors propose that the right number of labels to show depends on both the query and the reader, whereas prior work usually fixes this depth across queries and readers.

The task is multi-intent detection using a frozen LM through prompting, which makes this prompt engineering and in-context exemplar selection. The approach is a retrieve-and-rank framework applied to multi-intent detection from query utterances. The pool depth is the size of the retrieved top-k intent set that is passed to the frozen LLM. This LLM, called the reader, plays the role of the more accurate but slower ranker often used in retrieval systems.

The authors factorize the problem as P(exact) = P(gold covered | x) × P(exact | covered, x, m). The first term is a retrieval property and the second is a reader property. This is a very common factorization, also used in retrieve-then-rerank frameworks in large-scale retrieval. The benefit is that the individual terms can be sufficiently well approximated by choosing a subset of the intent classes instead of using the full set. However, doing so can also bring both accuracy and latency benefits when modeling the second factor with practical neural networks.

Their design differs from more traditional approaches through two meta-models that goes beyond traditional retrieve-and-rerank methods just learning to approximates the factors and also learns estimators for the probabilities themselves for a chosen cover size.

  1. The selector network: Given a query and a schema, the selector predicts the probability distribution over the minimum gold cover rank, which is the smallest depth that contains the gold set of intents for a given query for an encoder model. Coverage comes from this selector, which is trained on source domains.
  2. Reader "capacity"/tolerance for additional intents: meaning how many extra intents the language model tolerates, comes from label-free synthetic self-tests built from target descriptions.
Critical strengths
  1. Overall the work is well done. Lots of baselines, model settings and configurations are covered, so generality is strong.
  2. The motivation is very clear and properly established using experiments (Fig. 1).
  3. One key strength is that, in this approach the context size is no longer a hyperparameter; it is part of the model's inference.
Critical weaknesses
  1. Novelty relative to retrieve-and-rerank: While the mathematical framing is elegant, the idea of retrieve and rerank is not new and is very common in most large-scale retrieval systems. Whether this specific framing has any advantage over a standard retrieve-and-rerank pipeline remains to be seen. Two additional models are trained here: one for the gold minimum coverage rank and one for the reader's tolerance for additional intents. Adding two more trained models to the decision-making pipeline introduces additional bottlenecks and the risk of accumulated errors. The paper has not tested whether a simpler pipeline with heuristic-based but more robust rules for dynamic exposure setting could achieve the same or better performance.

  2. Reliance on failure prediction: Both the selector and the difficulty predictor are trying to predict the performance or failures of a language model. This is known as error prediction or failure prediction, where a model is trained to predict when it cannot make the right prediction. It is a difficult problem, and such predictors often don't generalize outside the training distribution. A learned error predictor is just another model trained on the same distribution, so it inherits the same blind spots. Under shift, the inputs where the base model fails are often exactly where the error predictor is also extrapolating (Jaeger et al., https://arxiv.org/pdf/2211.15259).

  3. Encoder dependence: As per Equation 2, the definition of the minimum gold cover rank depends on the query and intent encoder models. This means the selector training depends on the choice of a specific encoder model.

  4. Research gap and model choice: What is the real research gap being addressed? The authors argue for adapting exposure in schema-driven intent prediction with frozen language models. Given that these models are becoming exceedingly capable, how much leeway does adjusting the prompt still provide? Have you tried your approach with newer LLMs at the same scale as the Qwen3 models in the paper (which are quite dated now), or with larger, more capable models? In both cases, I would strongly recommend testing with up-to-date models.

  5. Datasets: These datasets (ATIS, SNIPS, Banking77, etc.) are very old and were very likely seen by the models during training, so they are not really out-of-domain. This raises a lots of questions on the validity of the experiments and the conclusions.

Questions for authors
  1. "A new intent domain often provides label names and descriptions before labeled utterances." What is the point of this statement?
  2. The initial intent retrieval stage depends on a sentence encoder, which is not as powerful as the LLM and is therefore a performance bottleneck. Why not use the LLM itself as the sole classifier and remove the pool selection step? We are already making an API call to the LLM, so increasing the prompt size to use the LLM as the sole intent classifier should not take significantly more time.
  3. How is the pre-trained sentence encoder expected to bring queries and intent descriptions closer together? Queries and intent descriptions do not have similar meanings; an intent description describes a particular class of queries and not the meaning of the query itself. That is not how embedding models are trained.
  4. Why did you choose "all-mpnet-base-v2" as the encoder?
  5. What does the subscript S indicate in Equation 4 for the variable I_S?
  6. CAPE uses 3 calls and about 2,017 tokens per query (Full: 1,869). How does this translate into real API calls and costs?
2 comments

Thank you for the detailed questions. We distinguish below between evidence already included in the paper and analyses we have now added in response to the review. Each answer identifies the relevant table, figure, or section; manuscript changes are marked in blue.

1. What is new relative to retrieve-and-rerank, and how does CAPE compare with simpler exposure rules?

The contribution is a way to measure a deployed frozen reader’s tolerance of additional labels and use that measurement to select exposure without target annotations or reader internals. The coverage–preservation factorization organizes the problem; we do not claim the factorization itself as a new retrieval principle. CAPE keeps the relevance ranking fixed and chooses how much of it to expose. The reader predicts an intent set rather than merely reranking candidates.

The requested comparisons with simpler exposure rules are already included in the paper: Adaptive-k in Table 1, and matched calibrated policies in Tables 2 and 17. Table 17 includes synthetic-selected fixed K, coverage thresholding, and CAPE basic; basic removes Fisher/EVI selection and the query-level update. Under matched calibration budgets and three query calls, Table 2 reports CAPE’s advantages of +0.86 percentage points (pp) over thresholding and +0.49 pp over basic, with paired 95% CIs [0.61, 1.15] and [0.22, 0.78]. These comparisons were therefore part of the evaluation before this revision; their numerical results are unchanged. We have now made their connection to the contribution more explicit in the Introduction and Section 6.

2. Can the two trained components accumulate errors, particularly under distribution shift?

Yes, coverage-transfer errors can propagate to the final decision. However, the two trained components and deployed-reader calibration have different roles. As described in Section 4.2 and Appendix C.2, the source response model designs initial self-tests, while the deployed reader’s capacity posterior is inferred from that reader’s own self-test outcomes. It is not simply the output of a fixed source-trained failure predictor applied to an unseen reader. This distinction is part of the method described there; we have now emphasized it in Section 4.2.

The paper already evaluates transfer to unseen readers in Figure 4 and Tables 22–23: six readers, including the Llama family, are excluded from source response-model fitting. Relative to the empirical reference, held-out reader-rank correlation increases from 0.620 to 0.762, and curve MAE decreases from 0.231 to 0.188. These existing results directly test transfer of the behavioral measurement.

To extend that evidence, we have now added failure and stability checks. Appendix Q.1, Tables 32–35 analyzes ATIS coverage overprediction; gold omission accounts for 86.8% of the Full-correct-to-CAPE-wrong transitions for Qwen3-4B/8B/14B/32B. Appendix Q.2, Tables 36–37 adds an input-only Full switch that improves ATIS Exact from 37.46 to 38.88 and the 72-setting mean from 44.91 to 45.12. Appendix Q.3, Table 38 regenerates three probe banks and finds positive paired gains over both matched controls in every repeat of the two-reader audit. These results identify a concrete transfer failure and test a source-calibrated response to it, without assuming that OOD errors disappear.

3. Why use MPNet, are query–description embeddings appropriate, and is the method encoder-dependent?

We agree that R is defined relative to the encoder ranking. Section 4.1 and Appendix B.1 specify MPNet as the frozen reference encoder; keeping this ranking shared across policies isolates the exposure decision. Its role is to provide semantic relevance, not to assume that a category description is a paraphrase of a query. To assess the resulting retrieval more directly, we have now added Table 32, reporting complete-gold coverage, including 68.4% at K=5 and 88.7% at K=10 on ATIS.

We have now added an encoder-replacement experiment in Appendix Q.8, Table 47 to test whether the policy benefit is specific to MPNet. We replace MPNet with BAAI/bge-base-en-v1.5, using CLS pooling, L2 normalization, and no query instruction prefix, and refit encoder-dependent source components under the same domain holdouts. On Qwen3-8B and Llama-3.2-3B, both BF16, across ATIS/CLINC150 and Blend/Mix, threshold/basic/CAPE achieve 44.91/45.34/45.85 Exact, respectively: +0.94/+0.51 pp for CAPE. The policy advantage therefore extends to this replacement. We retain encoder dependence as a limitation; this experiment does not establish MPNet’s optimality or remove retrieval as a possible bottleneck.

4. Does exposure adaptation remain useful for newer or larger readers?

The evaluation already includes larger and hosted readers: Qwen3-32B in Table 1 and two hosted readers in Table 4, with both favorable and unfavorable outcomes reported. In response to the request for a newer reader, we have now added Qwen3.5-4B, BF16, non-thinking mode, on all four domains under both suites.

Appendix Q.6, Table 45 reports macro Exact of 59.15/59.49/59.78/60.28 for Full/threshold/basic/CAPE. CAPE gains +0.79 pp over threshold and +0.50 pp over basic. Mix ATIS remains unfavorable, so the result supports a matched-policy benefit on the additional reader without claiming uniform gains. Section 6 now summarizes this experiment.

5. Do old benchmarks invalidate the OOD claim because of possible pretraining exposure?

Our OOD claim concerns transfer of the exposure method to a domain excluded from its training and model selection. The leave-one-domain-out protocol is already specified in Section 5: the other three domains supply source training and validation. This does not establish that the foundation models never encountered these benchmarks during pretraining. Section 7 now explicitly distinguishes domain holdout for our method from verified exclusion from foundation-model pretraining. The conclusions are scoped to the former.

6. Why not use the LLM alone with the complete schema?

The paper already includes exactly this comparison as the Full baseline in Tables 1 and 4: all intent descriptions are provided in one reader call, without pool truncation. Tables 11–12 already give the detailed outcomes, including cases where Full wins. The complete-schema classifier was therefore evaluated throughout; it is not a baseline added only for this revision.

The question is when restricting exposure improves a particular reader's predictions. Section 6 now connects that choice to measured accuracy and latency and describes the optional input-only Full switch. Full remains the one-call option; CAPE is not presented as a universally better replacement.

7. Clarifications of the deployment scenario and notation.

(a) Descriptions before labeled utterances. The Introduction refers to a service whose intent taxonomy and descriptions are defined before collecting annotated utterances for that domain. Unlabeled deployment utterances may still be available. Their use for calibration-call allocation is already specified in Section 5 and Appendix D.2. We have clarified the deployment example in the Introduction.

(b) The subscript SS in Eq. 4. ISI_S is the source/training-domain schema, distinct from the held-out target schema. Section 4.1 now defines SS explicitly immediately before Eq. 4.

8. What do three calls and the reported tokens mean for actual runtime and cost?

The three-call accounting is already specified in Section 4.3 and Table 5: two order-view calls followed by one final prediction. The 2,017 amortized tokens/query reported in Table 5 already account for these calls, versus 1,869 for Full.

We have now added measured runtime and separate preparation costs in Appendix Q.5, Tables 41–44, to complement that call/token accounting. The audit runs on RTX 5090 with BF16 weights, batch size/concurrency one, and sequential calls:

ReaderFull mean latencyBasicCAPECAPE vs. basic
Qwen3-8B214.5 ms482.1 ms491.4 ms+9.3 ms (+1.93%)
Llama-3.2-3B128.4 ms276.8 ms281.5 ms+4.7 ms (+1.70%)

These are end-to-end times. Tables 41–42 separately report input/output tokens summed across calls; CAPE uses 2,638/34.1 and 1,596/31.6 tokens/query for the two readers. Tables 43–44 report one-time preparation separately, taking 40.75/24.85 seconds for CAPE in this audit.

For API billing, the accounting is input-token charges plus output-token charges across all calls. Tables 41–44 provide the corresponding token counts, and Appendix K specifies how one-time preparation is amortized across deployment queries. We provide these measurements and local timings rather than an unmeasured provider-specific dollar or latency estimate. CAPE's total latency is 2.29/2.19 times Full, while its incremental execution cost over basic is much smaller; Section 6 now makes this trade-off explicit.

We hope this clarifies that the simpler-policy comparisons, full-schema baseline, and held-out-reader evaluation were already included, while the revision adds the requested encoder, reader, failure, and runtime evidence. We would appreciate your reassessment in light of both.

Comments on the paper

2 comments

Summary of revisions and responses

We thank all three reviewers for their careful feedback. Our point-by-point responses address each concern and identify the corresponding evidence in the paper. We have revised the manuscript, with changes marked in blue, and summarize the main clarifications and additions below.

Evidence already included in the paper. The evaluation already contained the full-schema baseline (Tables 1 and 4), simpler exposure policies and matched calibrated controls (Tables 1, 2, and 17), and transfer evaluation on six held-out readers (Figure 4 and Tables 22–23). Our responses identify these comparisons explicitly and explain how they address the relevant questions. We also clarify that reader calibration provides the larger benefit, while the complete CAPE procedure provides a smaller additional gain over the matched calibrated controls.

New evidence added in response to the reviews:

  • ATIS failure diagnosis and a label-free Full-selection rule — Appendix Q.1–Q.2, Tables 32–37. We analyze retrieval coverage, coverage overprediction, and prediction transitions. We then evaluate an input-only switch whose threshold is calibrated exclusively on source data and applied before query-level reader calls. It improves ATIS Exact by +1.42 pp [95% CI: +0.94, +1.91] and the 72-setting mean by +0.21 pp [+0.04, +0.38]. The reported domain breakdown also makes its limitations explicit.

  • Recalibration variability and small/shifted buffers — Appendix Q.3–Q.4, Tables 38–40. Across three independently regenerated probe banks in the two-reader audit, CAPE retains positive gains over both matched controls in every repeat: +0.78±0.05+0.78 \pm 0.05 pp over threshold and +0.67±0.05+0.67 \pm 0.05 pp over basic. Separate experiments exclude evaluation queries from the allocation reservoir and test both small random buffers and a specified label-free distribution shift.

  • Measured execution and preparation costs — Appendix Q.5, Tables 41–44. We report end-to-end latency, input/output tokens, and one-time preparation separately. CAPE adds 9.3 ms and 4.7 ms over basic for the two audited readers, while taking 2.29 and 2.19 times Full's latency. Section 6 now connects these measured trade-offs to the choice among Full, threshold, basic, and CAPE.

  • An additional reader and an alternative encoder — Appendix Q.6 and Q.8, Tables 45 and 47. With Qwen3.5-4B in non-thinking mode, CAPE improves over threshold/basic by +0.79/+0.50 pp across eight settings. Replacing MPNet with BGE and refitting encoder-dependent source components yields +0.94/+0.51 pp over the same controls in the eight-setting encoder audit.

  • Figure interpretation and claim precision. We clarify Figure 1's markers and the distinction between best tested depth and reliable schema use, supported by the new output analysis in Appendix Q.7, Table 46. The Abstract now distinguishes percentage-point and relative improvements, and Section 7 clarifies the scope of the domain-holdout claim.

We would appreciate the reviewers' reassessment in light of these clarifications and additional results. We welcome any remaining questions.

AuthorsOpenReview ↗

Gentle Reminder to All Reviewers

Dear reviewers,

Thank you again for your constructive feedback. We have responded directly to every concern raised in the reviews and revised the manuscript accordingly. Before the discussion closes, we would appreciate a brief note on any remaining concerns we can address.

Best, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.