- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies cost-constrained classification of customer-call summaries into more than 100 long-tailed categories. The evaluated system uses a fine-tuned DeBERTa encoder to answer high-confidence calls and routes the remaining calls to an LLM with a fixed top-5 candidate list. The authors also formulate a Conformal Shortlist Cascade (CSC), in which a calibrated prediction set jointly determines escalation and the candidate labels exposed to the LLM. Experiments examine prompt design and candidate recall, reporting that a sequence of shortlist-specific prompt modifications improves conditional accuracy from 42.7% to 63.5% on a small human-gold hard slice. The paper reports over 90% cost savings relative to invoking the LLM with the full taxonomy on every call. However, the evaluated operating point uses fixed routing rather than an experimentally validated adaptive conformal policy.
Critical strengths
-
Practically important setting. The paper addresses the cost of classification over a large, long-tailed taxonomy at industrial volume. Separating cheap encoder decisions from expensive LLM adjudication makes the operational trade-off concrete and relevant to deployed classification systems.
-
Useful analysis of candidate recall and prompt adaptation. Evaluating picker accuracy conditional on the true label being present helps distinguish candidate-generation failures from selection failures. The prompt study and encoder-versus-retrieval comparison provide potentially useful evidence that candidate descriptions and ranking quality matter on the routed hard cases.
Critical weaknesses
-
The distinction from prior work is not sufficiently established.
-
Dual-Expert (Cheng et al., 2024). Section 2 describes this work as having no accept/escalate gate, although its Section 4.3.1 routes only approximately 20% of traffic through the LLM. Appendix A.2 of the current paper acknowledges this gate, contradicting Section 2. Dual-Expert also combines a fine-tuned encoder, top-k candidates, LLM selection, and definitions synthesized from examples, evaluated in its Section 4.2.3. Both the deployed architecture and a central prompt intervention therefore have close precedents.
-
Conformal Cascade (Dou et al., 2026). Section 2 and Appendix A.5 explain that this work uses a conformal set to decide escalation but does not forward that set as the downstream candidate list. CSC proposes to reuse the set for candidate restriction and provide per-frequency-band coverage. This difference is stated, but the fixed-routing experiments do not establish its contribution.
-
CROQ (Vishwakarma et al., 2025). Section 2 and Appendix A.5 describe CROQ as using a conformal set to restrict the LLM's options without a cheap-model termination stage. CSC proposes to use the same set for both termination and candidate restriction. However, the evaluated fixed-threshold, top-5 system does not validate this joint conformal mechanism.
-
-
The experiments do not fully establish the method's contribution.
-
The evaluation does not validate CSC's coverage claims. The authors acknowledge in Section 11 that the experiments evaluate fixed routing rather than adaptive-conformal superiority. Yet Sections 4 and 11 claim that realized per-band coverage is reported, while Appendix A.8 states that this breakdown and the conformal set-size distribution are not produced. The experiments therefore do not validate the proposed joint calibration or its long-tail coverage in deployment.
-
The recall figures are inconsistent or insufficiently specified. Section 5 reports over 90% recall on the human-gold hard slice, whereas Section 8 and Table A.2 report 80%. Tables 3 and A.3 give 95% for the escalated tier without identifying the reference labels or population that explains the difference. This ambiguity affects the reported conversion from conditional picker accuracy to path accuracy.
-
The shortlist-size conclusions rely on unmeasured accuracy changes. Tables 3 and A.3 state that top-3 and top-10 were not separately run and are projected from an input-token cost model. They nevertheless report accuracy changes of -5 and +1 percentage points, which Section 9 uses to assess shortlist sizing. A token-cost model alone cannot support these accuracy estimates, so the stated accuracy-cost trade-off lacks empirical support.
-
Questions for authors
-
Can you provide the supplementary materials stated in the paper? The package described in the reproducibility statement on page 9 is absent from the submitted materials.
-
How does the method differ from prior work? Please compare the evaluated system with Dual-Expert (Cheng et al., 2024) in routing, candidate generation, and category-definition synthesis, and reconcile the gate descriptions in Section 2 and Appendix A.2. Please clarify how CSC differs from Conformal Cascade and CROQ in escalation, candidate restriction, and coverage guarantees.
-
What experimental evidence supports CSC's contribution? Please compare CSC with fixed routing at matched cost and report overall human-gold accuracy, per-band coverage, and conformal set-size distributions. Please reconcile the coverage statements with Appendix A.8, specify the populations and reference labels behind the over-90%, 80%, and 95% recall figures, and provide the empirical basis for the top-3/top-10 accuracy changes.