- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The authors ask whether bounded decisions, such as intent routing or multiple-choice selection, are cheaper to serve without token-by-token generation. They compare generation, constrained decoding, likelihood scoring and several encoder-based heads behind one decision interface, measuring quality on four datasets and cost on one GPU.
Critical strengths
- The paper compares full decisions on identical requests, counting setup and host work rather than only model time.
- The reporting is very honest. Invalid outputs are kept, post hoc comparisons are labelled as such, and the limitations section clearly says that no comparison isolates the architecture.
Critical weaknesses
- The main cost comparison is between two implementations rather than two approaches. As the authors acknowledge, the two sides differ in engine (vLLM vs. the authors' PyTorch code), backbone and session (collected about 24,000 s apart), and only the decoder had its per-request check moved out of the timed call, so the query head still tokenizes inside it. The query head also encodes every candidate description as a separate sequence on each call, 77 per request on BANKING77, even though this label set is fixed and could be encoded once. In the paper's own one-factor cells the ordering already reverses at batch 32 (heads at 439.4 and 446.7 vs. decoder at 316.9 in Table 8), so I don't think the abstract's claim that "eliminating answer generation does not by itself lower cost" is supported beyond these two implementations.
- The other main results add less than the abstract suggests. The 13× speed-up in Table 2 comes from moving a check out of the timed window. As the authors themselves note, certification plus one call adds up to 1.290 s vs. 1.233 s before (l.404), so there is no first-use gain, and idle-adjusted energy barely changes (0.961 vs. 0.957 J). This is an accounting change rather than a finding, yet the abstract presents it as "the largest measured change". The learned heads are also too weak for a quality-matched cost comparison: on the SciQ cohort used for timing, the query head reaches 0.474 against 0.994 for the decoder it is compared with in Table 6, and on BANKING77 0.733 vs. 0.804.
Questions for authors
see weaknesses
3 comments
Thank you for the matched study and the revised claims. Most of my concerns have been addressed. However, the classifier's first-use advantage on BANKING77 seems to come mostly from the decoder's ~37 ms per-request token-boundary check. Since the query head may encode the fixed 77 intents once, could you elaborate on why this check cannot be amortized in the same way, or am I misunderstanding something?
Thank you. You are not misunderstanding: the check can be amortized in our setting, and we will correct the first-use accounting.
Our implementation runs the check per request. It tokenizes the full prompt, which contains the request and all 77 choices, once on its own and once with each of the 77 identifiers appended, and compares the results. This is what costs about 37 ms per request. In our prompt, however, the identifier always follows the same chat-template suffix, which ends with special tokens, so the tokens at the boundary do not depend on the request. The check therefore needs to run only once per identifier set and template, in the same way that the query head encodes the 77 intents once. Per request, the decoder then pays only one prompt tokenization.
With the check amortized, the relevant BANKING77 comparison is the one with prepared requests (Figure 2, Table 1). At batch size 32 the classifier completed 7.9x more decisions per second than the decoder (3,166 versus 401, accuracy 0.905 versus 0.804); at batch size 1 the two were not separated (159 versus 168). The classifier's batch-32 advantage therefore does not depend on the check. Its first-use advantage at batch size 1 (158 versus 23) does, and so does the first-use reversal of the query-head ordering on BANKING77 at batch size 1.
In the final version, first-use accounting will charge the decoder one prompt tokenization per request instead of the per-request check. The paper will no longer state a batch-1 first-use advantage for the classifier on BANKING77, and it will describe the per-request check as a property of our implementation, not of constrained decoding.
Thank you for a specific and constructive review. We ran the study you describe and rewrote the cost claim around it.
We agree. The submitted abstract sentence was not supported beyond those two implementations, and it is removed. The new matched study (Section 4.1, Figure 2, Table 1, Appendix B) removes the session and input-preparation confounds, while the engine and backbone differences remain (Section 5). One process holds the vLLM decoder and the PyTorch encoders on one GPU. Within each block all units run in a seeded random order, so the two sides alternate within minutes instead of running 24,000 s apart. Every model receives prepared token IDs. The decoder runs its token-boundary check at startup, and the BANKING77 query head encodes the 77 intents once. We made two runs per dataset at batch sizes 1 and 32, and all 260 blocks were clean. A direction is stated only when both runs agree. With prepared requests at batch size 32, the query head was 5.0x faster than the decoder on BANKING77 (accuracy 0.733 versus 0.804), while on SciQ the decoder was 1.7x faster (0.994 versus 0.474); at batch size 1 the query head was slower on both cohorts. Counting per-request preparation reverses the SciQ batch-32 and BANKING77 batch-1 query-head orderings (Table 1), so the main claim rests on the quality-matched comparisons in W5. We also re-measured the submitted pair in the same runs, and it keeps its original timed ordering (ratios 0.56 to 0.64 at batch size 32). Its preparation was unequal, so the matched comparison supersedes it (Appendix B).
This is correct. Encoding the 77 intent descriptions once raised query-head throughput 9.2 to 9.9x at batch size 32 and 1.6x at batch size 1. Decisions were identical on all 3,079 BANKING77 test requests (Section 4.1, paragraph 'Input preparation'). This reuse needs a fixed label set, so it does not apply to SciQ, where 990 questions carry 985 distinct option sets. Preparing token IDs changed any model's throughput by at most 1.26x.
We agree, and the matched study shows the same reversal on real requests. On BANKING77 at batch size 32, both the query head and the classifier were faster than the decoder. The revised paper removes 'eliminating answer generation does not by itself lower cost' and makes no claim about generation itself. Instead it reports a workload-dependent ordering for the tested implementations (Abstract, Section 4.1, Section 6). Section 5 lists the remaining limits: engine, backbone, one GPU and two cohorts.
We agree. The result is removed from the abstract and the contributions and is now an implementation note (Section 4.4, paragraph 'Token-boundary check outside the timed call'). The note reports the first-use total of 1.290 s versus 1.233 s before, so there is no demonstrated first-use gain. It also reports the near-unchanged idle-adjusted energy (0.961 to 0.957 J per decision). The same first-use accounting now runs through the new cost comparison (Table 1).
We agree. The matched study adds two stronger encoders from the quality evaluation, with no retraining.
The query head remains a secondary model, and its accuracy (0.733, 0.474) accompanies each of its ratios.