acceptodds
Under review as a conference paper at ICLR 2027

OpenSystemOne: Efficient Parallel Inference for Dynamic Structured Decision Models

10 views

Abstract

Machine-facing AI workloads often require selecting, scoring, or verifying a bounded set of alternatives rather than generating natural language. Yet such workloads are commonly served by autoregressive decoders, and whether their computational structure matches the task is rarely measured at the level of the decision. We present OpenSystemOne, an open benchmarking and systems framework for parallel structured decision inference. The framework implements and compares free and constrained autoregressive generation, option likelihood scoring with and without shared-prefix reuse, encoder-based classifiers, query-conditioned decision heads, and learned dynamic-option models under a common System One interface. We measure accuracy and calibration on four public decision benchmarks and probe reordered, reduced, gold-absent and out-of-scope candidate sets in intent routing. On one Blackwell GPU, we measure latency, throughput, device memory and GPU energy on fixed request cohorts and vary context length, option cardinality and batch size one factor at a time. Eliminating answer generation does not by itself lower cost: a one-token constrained decoder already decides after one prefill, and on fixed batch-32 cohorts it completes more decisions per second than our encoder query head (342 versus 224 on BANKING77). The largest measured change was host-side: moving a per-request tokenization check out of timed decoder calls raises throughput 13-fold for requests prepared in advance, but counting that setup leaves no demonstrated first-use gain. In FP32, the precision in which reused and fresh paths agree within tolerance, prefix reuse raises multi-question throughput at higher memory cost. Quality and calibration vary by predictor and workload, and changing candidate sets expose order sensitivity and constant abstention. Characterizing when eliminating token-by-token answer generation pays off end to end, we find that for the tested implementations cost resides in the complete execution path rather than in generated tokens, while quality on the served candidate sets decides which fast path is usable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI systems, structured inference, parallel inference, non-autoregressive decision models, GPU optimization, dynamic choices, inference serving, calibration, latency, throughput

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NR2mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The authors ask whether bounded decisions, such as intent routing or multiple-choice selection, are cheaper to serve without token-by-token generation. They compare generation, constrained decoding, likelihood scoring and several encoder-based heads behind one decision interface, measuring quality on four datasets and cost on one GPU.

Critical strengths
  • The paper compares full decisions on identical requests, counting setup and host work rather than only model time.
  • The reporting is very honest. Invalid outputs are kept, post hoc comparisons are labelled as such, and the limitations section clearly says that no comparison isolates the architecture.
Critical weaknesses
  • The main cost comparison is between two implementations rather than two approaches. As the authors acknowledge, the two sides differ in engine (vLLM vs. the authors' PyTorch code), backbone and session (collected about 24,000 s apart), and only the decoder had its per-request check moved out of the timed call, so the query head still tokenizes inside it. The query head also encodes every candidate description as a separate sequence on each call, 77 per request on BANKING77, even though this label set is fixed and could be encoded once. In the paper's own one-factor cells the ordering already reverses at batch 32 (heads at 439.4 and 446.7 vs. decoder at 316.9 in Table 8), so I don't think the abstract's claim that "eliminating answer generation does not by itself lower cost" is supported beyond these two implementations.
  • The other main results add less than the abstract suggests. The 13× speed-up in Table 2 comes from moving a check out of the timed window. As the authors themselves note, certification plus one call adds up to 1.290 s vs. 1.233 s before (l.404), so there is no first-use gain, and idle-adjusted energy barely changes (0.961 vs. 0.957 J). This is an accounting change rather than a finding, yet the abstract presents it as "the largest measured change". The learned heads are also too weak for a quality-matched cost comparison: on the SciQ cohort used for timing, the query head reaches 0.474 against 0.994 for the decoder it is compared with in Table 6, and on BANKING77 0.733 vs. 0.804.
Questions for authors

see weaknesses

3 comments

Thank you for a specific and constructive review. We ran the study you describe and rewrote the cost claim around it.

W1: between two implementations rather than two approaches

We agree. The submitted abstract sentence was not supported beyond those two implementations, and it is removed. The new matched study (Section 4.1, Figure 2, Table 1, Appendix B) removes the session and input-preparation confounds, while the engine and backbone differences remain (Section 5). One process holds the vLLM decoder and the PyTorch encoders on one GPU. Within each block all units run in a seeded random order, so the two sides alternate within minutes instead of running 24,000 s apart. Every model receives prepared token IDs. The decoder runs its token-boundary check at startup, and the BANKING77 query head encodes the 77 intents once. We made two runs per dataset at batch sizes 1 and 32, and all 260 blocks were clean. A direction is stated only when both runs agree. With prepared requests at batch size 32, the query head was 5.0x faster than the decoder on BANKING77 (accuracy 0.733 versus 0.804), while on SciQ the decoder was 1.7x faster (0.994 versus 0.474); at batch size 1 the query head was slower on both cohorts. Counting per-request preparation reverses the SciQ batch-32 and BANKING77 batch-1 query-head orderings (Table 1), so the main claim rests on the quality-matched comparisons in W5. We also re-measured the submitted pair in the same runs, and it keeps its original timed ordering (ratios 0.56 to 0.64 at batch size 32). Its preparation was unequal, so the matched comparison supersedes it (Appendix B).

W2: 77 per request on BANKING77

This is correct. Encoding the 77 intent descriptions once raised query-head throughput 9.2 to 9.9x at batch size 32 and 1.6x at batch size 1. Decisions were identical on all 3,079 BANKING77 test requests (Section 4.1, paragraph 'Input preparation'). This reuse needs a fixed label set, so it does not apply to SciQ, where 990 questions carry 985 distinct option sets. Preparing token IDs changed any model's throughput by at most 1.26x.

W3: the ordering already reverses at batch 32

We agree, and the matched study shows the same reversal on real requests. On BANKING77 at batch size 32, both the query head and the classifier were faster than the decoder. The revised paper removes 'eliminating answer generation does not by itself lower cost' and makes no claim about generation itself. Instead it reports a workload-dependent ordering for the tested implementations (Abstract, Section 4.1, Section 6). Section 5 lists the remaining limits: engine, backbone, one GPU and two cohorts.

W4: an accounting change rather than a finding

We agree. The result is removed from the abstract and the contributions and is now an implementation note (Section 4.4, paragraph 'Token-boundary check outside the timed call'). The note reports the first-use total of 1.290 s versus 1.233 s before, so there is no demonstrated first-use gain. It also reports the near-unchanged idle-adjusted energy (0.961 to 0.957 J per decision). The same first-use accounting now runs through the new cost comparison (Table 1).

W5: heads too weak for a quality-matched cost comparison

We agree. The matched study adds two stronger encoders from the quality evaluation, with no retraining.

  • BANKING77: the fixed-label classifier is more accurate than the decoder (0.905 versus 0.804). With prepared requests it completed 7.9x more decisions per second at batch size 32 (3,166 versus 401), and at batch size 1 the two were not separated (159 versus 168). Counting the decoder's per-request token-boundary check (about 37 ms), the classifier is faster at both batch sizes (158 versus 23 and 2,676 versus 25 decisions/s).
  • SciQ: the cross-encoder matches the decoder's 0.994, and the decoder was faster at both batch sizes. With prepared requests it was 7.0x faster at batch size 32 and 1.2x at batch size 1. In first-use accounting the ratios are 3.5x and 1.1x.

The query head remains a secondary model, and its accuracy (0.733, 0.474) accompanies each of its ratios.

Thank you for the matched study and the revised claims. Most of my concerns have been addressed. However, the classifier's first-use advantage on BANKING77 seems to come mostly from the decoder's ~37 ms per-request token-boundary check. Since the query head may encode the fixed 77 intents once, could you elaborate on why this check cannot be amortized in the same way, or am I misunderstanding something?

Thank you. You are not misunderstanding: the check can be amortized in our setting, and we will correct the first-use accounting.

Our implementation runs the check per request. It tokenizes the full prompt, which contains the request and all 77 choices, once on its own and once with each of the 77 identifiers appended, and compares the results. This is what costs about 37 ms per request. In our prompt, however, the identifier always follows the same chat-template suffix, which ends with special tokens, so the tokens at the boundary do not depend on the request. The check therefore needs to run only once per identifier set and template, in the same way that the query head encodes the 77 intents once. Per request, the decoder then pays only one prompt tokenization.

With the check amortized, the relevant BANKING77 comparison is the one with prepared requests (Figure 2, Table 1). At batch size 32 the classifier completed 7.9x more decisions per second than the decoder (3,166 versus 401, accuracy 0.905 versus 0.804); at batch size 1 the two were not separated (159 versus 168). The classifier's batch-32 advantage therefore does not depend on the check. Its first-use advantage at batch size 1 (158 versus 23) does, and so does the first-use reversal of the query-head ordering on BANKING77 at batch size 1.

In the final version, first-use accounting will charge the decoder one prompt tokenization per request instead of the per-request check. The paper will no longer state a batch-1 first-use advantage for the classifier on BANKING77, and it will describe the per-request check as a property of our implementation, not of constrained decoding.

Reviewer_9cJvOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

If I'm understanding this paper correctly, this paper is discussing a dynamic system to serve the LLMs, including routing the prompts, scheduling the tasks etc, to fully utilize the compute resource.

Critical strengths

The paper looks sound and cover a comprehensive study across different factors of serving models efficiently. I don't really understand too much details but it looks solid and sound.

Critical weaknesses

It is not necessarily a weakness, and depends on how authors define the audience of the paper. I personally think this paper is hard to read as a model arch/scaling/algo researcher.

Questions for authors

No tech questions

2 comments

Thank you for reviewing a paper outside your area and for the candid comment.

S1: a dynamic system to serve the LLMs

The paper is not a scheduling or routing system for LLM serving. It compares ways of answering a bounded decision, which means picking one of K known options, such as one of 77 banking intents or one of four answers. One way is a decoder that emits one constrained token, and another is an encoder that generates nothing. We measure accuracy and serving throughput for each on identical requests.

W1: hard to read as a model arch/scaling/algo researcher

We agree and rewrote the paper for model and algorithm readers. Figure 1 shows how each model family turns the same request into the same decision. The introduction states one question and one main finding (Section 1), and one figure shows the main result (Figure 2, Section 4.1). At equal or higher accuracy, the faster model depended on the workload. For 77 fixed intents, an encoder classifier was 7.9x faster than the decoder. For four-option science questions, the decoder was 7.0x faster than an equally accurate cross-encoder. Both ratios are at batch size 32 with requests prepared before timing. Internal jargon is removed. Thank you again.

Thanks for the reply. I still decided to keep my score and my confidence as 1 (very low confidence) due to the lack of background to understand this work.

Reviewer_36jiOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents OpenSystemOne and a series of results ablating different configurations on BANKING77 and CLINIC150 benchmarks.

Critical strengths

None

Critical weaknesses

The paper is unintelligible. It is unclear if it's an engineering work contributing a framework or a method proposal for constrained decoding. It presents to be a series of experiments but the methodology is poorly described and it is unclear what is the research question or the hypotheses under test. It is also unclear how the experiments are related The work seems to be an implementation of existing work but unclear what is different about the implementation from existing work. The writing uses undefined jargon "2x2 separation of batching", "decision contract", "natural-input cells", "Replicate stability" that makes understanding difficult. There are no evaluation against any baselines, only different configurations of the framework. It is also unclear what is being changed or the significance.

Questions for authors
  • What models exactly are System One models?
  • Could the authors explain what "Natural-input cells (42) measure 21 endpoint identities at batch size 1 and 32 on two 32-request cohorts fixed by request identifiers before collection" mean? And what it means that "The answers are conditional, and the conditions carry the main lesson"?
  • What exactly is being profiled?
  • What are the baselines? How does the work compare against Jev, VLLM, SGLang? How are they related?
1 comment

Thank you for the direct review. We agree that the submitted version did not state its question or connect its experiments, and we rewrote the paper around both.

W1: unclear what is the research question

The paper is an empirical comparison on one open interface. It does not propose a new decoding method or a serving framework. The question (Section 1): for bounded decisions such as intent routing or multiple-choice selection, does a model that generates nothing serve decisions faster than a one-token constrained decoder at matched quality? The experiments build on each other. Section 4.1 answers the question with a new matched timing study (Figure 2, Table 1, Appendix B), using the accuracies measured in Section 4.2; Section 4.3 tests changing candidate sets. Figure 1 shows how each model family handles the same request. The answer is that, at equal or higher accuracy, the faster model depended on the workload.

W2: an implementation of existing work

We agree that the components predate us, and Sections 1 and 2 now say so. The new contributions are two (Section 1). The first is a matched serving comparison of a one-token constrained decoder with three encoder models on identical requests. It matches input preparation, interleaves measurements in one process and reports accuracy next to throughput and GPU energy. The second is the open interface, which runs all seven families with one decision format so that others can repeat such comparisons with open weights.

W3: undefined jargon

Each term is replaced:

  • 'decision contract' is now decision interface (Section 3).
  • 'natural-input cells' is now measurements on real test requests (Appendix A).
  • '2x2 separation of batching' is now batching versus prefix reuse, the four ways to answer many questions about one text (Section 4.4).
  • 'replicate stability' is now variation across training seeds (Section 4.2).
  • In the main text, 'endpoint' is now model-dataset configuration and 'certification' is now token-boundary check.

W4: no evaluation against any baselines

A factual correction: the submitted version already had baselines. The baseline that every encoder must beat is a one-token constrained decoder (Qwen3-0.6B with LoRA) served by vLLM. The quality comparison, also in the submission, covers free generation, likelihood scoring with and without prefix reuse, released Qwen3-0.6B, 1.7B and 4B, and released GLiClass-modern (Section 4.2, Appendix C). The matched study adds two stronger encoder baselines without retraining. The fixed-label classifier is more accurate than the decoder on BANKING77 (0.905 versus 0.804), and the cross-encoder is as accurate on SciQ (0.994). The comparisons vary the model family and its input preparation. Their significance is that the faster model at matched quality differed between the two workloads.

Q1: What models exactly are System One models?

A System One model returns a typed decision directly, the chosen candidate with a probability for each candidate, instead of text (Section 1). The compared models all run behind this interface (Section 3):

  • Qwen3-0.6B, released or adapted with LoRA, used for free generation, one-token constrained decoding and likelihood scoring.
  • Fully fine-tuned ModernBERT-base with four heads: fixed-label classifier, cross-encoder, query head and set head.
  • Released GLiClass-modern.

Q2: Natural-input cells (42)

We timed 21 trained or released models, each with its serving configuration, at batch sizes 1 and 32, giving 42 measurements. They were timed on two fixed groups of 32 test requests, BANKING77 in-scope messages and SciQ questions with passages. We chose the requests by identifier before any measurement, so every model serves identical requests (Section 3, Appendix A). The sentence about conditions is removed, and the finding is now stated explicitly (Section 4.1).

Q3: What exactly is being profiled?

We profiled host-side Python work. Python cProfile recorded 10 identical calls after one warm-up in two BANKING77 serving paths at batch size 32, the constrained decoder and the query head. It ranked functions by their own wall-clock time and does not see GPU kernels. The top item was the decoder repeating its per-request token-boundary check inside every call. Moving the check to startup is now an implementation note (Section 4.4, protocol in Appendix A).

Q4: Jev, VLLM, SGLang

vLLM serves the decoder baseline. SGLang is an alternative engine for the decoder side. We did not integrate or benchmark it and list it as future work (Section 5). Jev is a proprietary hosted model that we cannot run on our GPU. Our query and set heads share only its typed-decision interface, not its weights, architecture or training (Section 2).

Comments on the paper

1 comment

We thank the three reviewers for their time and for comments that changed the paper substantially. All locations refer to the revised PDF. What changed One research question now organizes the paper. For bounded decisions such as intent routing or multiple-choice selection, does a model that generates nothing serve decisions faster than a one-token constrained decoder at matched quality? The title, abstract, introduction and conclusion are rewritten around it (Section 1, Section 6). The paper is rewritten for readers outside serving systems, with new figures and defined terms (see the summary of changes below). A new matched timing study answers Reviewer NR2m (Section 4.1, Appendix B). The submitted 13x latency reduction came from moving the token-boundary check out of the timed call. It is now an implementation note (Section 4.4). The note keeps its first-use accounting, which shows no demonstrated first-use gain. Main finding At equal or higher accuracy, the faster tested model depended on the workload. For the 77 fixed BANKING77 intents, an encoder classifier (accuracy 0.905) was 7.9x faster than a one-token constrained decoder (0.804) at batch size 32 with requests prepared before timing. For four-option SciQ questions, the decoder was 7.0x faster than an equally accurate cross-encoder (0.994) at batch size 32 with prepared requests, and 3.5x when per-request preparation is counted. At batch size 32, both orderings hold with and without counting per-request preparation. The new experiment One process holds the vLLM decoder and the PyTorch encoder models on one GPU. Within each measurement block, every combination of model, preparation and batch size runs in a seeded random order. Decoder and encoder measurements therefore alternate within minutes; in the submission they were collected about 24,000 s apart. Each model gets the best input preparation available without retraining. All models receive token IDs prepared before serving, and the BANKING77 query head encodes the 77 intent descriptions once. Two stronger encoders join the query head: the fixed-label classifier on BANKING77 and the cross-encoder on SciQ. We made four measurement runs, two per dataset, at batch sizes 1 and 32 on the fixed 32-request cohorts of the original cost evaluation. All 260 measurement blocks were clean, and a direction is reported only when both runs agree. No model was retrained. On the full test sets, the matched encoder variants select exactly the same answers as the submitted implementations (3,079/3,079 BANKING77 and 990/990 SciQ requests, maximum probability difference 0). We report timed throughput for requests prepared before timing. We also report a first-use accounting that adds per-request preparation to each decision, mainly the decoder's token-boundary check of about 37 ms per BANKING77 request (Table 1). Limitations The decoder and the encoders still differ in serving engine (vLLM versus our PyTorch code) and backbone (Qwen3-0.6B with LoRA versus ModernBERT-base), so no result isolates the effect of generating or not generating. The cost evidence covers one GPU and two cohorts. SGLang and hosted models such as Jev are not benchmarked (Section 5).

Summary of changes (revision round 1) Framing: title, abstract, introduction and conclusion rewritten around one research question (Sections 1, 6); the contributions list says the components predate this work (Sections 1, 2). Figures and tables: concept figure (Figure 1); timed throughput against accuracy (Figure 2); throughput in both accountings, prepared requests and first use (Table 1). Jargon removed from the main text: decision contract, natural-input cells, 2x2 separation of batching, replicate stability, endpoint, certification. Matched timing study (Section 4.1, Appendix B): as described above; the submitted query-head versus decoder pair was re-measured in the same runs and is superseded by the matched comparison. Claims: removed "eliminating answer generation does not by itself lower cost"; the new main claim is the workload-dependent ordering above; the 13x token-boundary result is an implementation note (Section 4.4); the query head is secondary, with its accuracy next to each ratio. Limitations (Section 5): adds that a choices-first decoder prompt with prefix caching would need retraining and is untested. Supplementary material: matched-study code, configurations, tests, result summaries and the first-use accounting script. Unchanged: quality, calibration and changing-candidate results; no model was retrained (Sections 4.2, 4.3; Appendices C, D).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.