acceptodds
Under review as a conference paper at ICLR 2027

SystemOneBench: A Cross-Domain Benchmark for Calibrated Machine-Native Intelligence

10 views

Abstract

Machine intelligence is increasingly embedded in software that requires structured decisions rather than open-ended text generation. Existing evaluations mostly measure generative quality, reasoning accuracy or task completion, and give limited evidence about the calibration, efficiency and reliability of models used as machine-consumed decision functions. System One interfaces, now offered as hosted services, serve this use by returning a probability distribution over a supplied answer space; this guarantees that every answer is well formed, not that it is informative or calibrated. We introduce SystemOneBench, a cross-domain benchmark protocol for calibrated structured decision models, instantiated on seven data families spanning financial market microstructure, blockchain transaction dynamics, road traffic, Earth-system observations, maritime operations and autonomous-agent traces. Each task maps a real-world state and a dynamically defined answer space to typed probabilistic decisions whose ground truth is derived from subsequently observed outcomes rather than model-generated labels. The protocol specifies paired, dependence-aware analyses of decision accuracy, proper scoring rules, calibration error, risk–coverage trade-offs, temporal and entity shift and option-set generalization, and a procedure for measuring latency, throughput and energy. Its harness presents free generation, constrained decoding and one-pass option scoring of one Qwen3-0.6B checkpoint, supervised references including learned decision heads, and a commercial System One model with identical states, answer spaces and outcomes. First results show that validity and information separate. Free and constrained generation return a valid distribution on every attempt, yet reproduce the prompt's uniform format example on all 15,491 outputs per readout; one-pass scoring never does. The hosted System One model is schema-valid on every agent-trace request, but its test log loss (0.7986 nats raw, 0.7747 after temperature scaling) exceeds that of a uniform prediction (0.6931); on maritime and weather tasks it assigns zero probability to eleven observed calibration events per family, leaving temperature scaling without a finite-loss solution. Machine-native decisions must be evaluated as probabilities on outcomes, not as outputs that fit a schema.

Primary area:
foundation or frontier models, including LLMs
Keywords:
calibrated decision models, System One models, machine-native intelligence, selective prediction, uncertainty calibration, dynamic label spaces, structured prediction, benchmark, distribution shift, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kUD4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a benchmark for calibration in "System One" models, for example like the recent release of Jev, a model by TypeSafe AI. These models provide typed outputs that are constraint to fit in a certain format. This paper presents a benchmark for the evaluation of calibration in these models. The benchmarks contains tasks across a set of 7 data families. There is some evaluation on Jev and Qwen3-0.6B.

Critical strengths
  • Thorough evaluation of these "System One" models to constrained decoding in LLMs is an important endeavor to understand their utility.
  • It does seem important to have benchmarks for these models.
Critical weaknesses
  • The writing in this paper is not easily comprehensible by humans. There is heavy jargon which is never clearly introduced and the as the reader it often feels that I am lacking context. The problem is not clearly motivated and the methods are not clearly explained. For example, without any background about the setup or details, this is a paragraph in the introduction: "The gap is not hypothetical. We elicited probability vectors from Qwen3-0.6B by free generation and by constrained generation that admits only valid vectors. Without a format example in the prompt, generation exhausted its output budget on every tuning input of a formatting pilot; with one, it completed"
  • The paper contains no figures.
  • The evaluation scheme is unclear and not comprehensive. All evaluations are on Qwen3-0.6B, which is too small to draw reasonable conclusions from and Jev, a closed source model without a public implementation. The only evaluation I can see is in Table 5, which is insufficient. There is no study across scales/models/architectures etc.
  • There is also an issue in that the tasks here are not fundamentally different to many tasks that are solved by current LLMs. It's not clear why these tasks should be used to evaluate "System One" models rather than many other benchmarks with a fixed output format. The key difference between these models and LLMs seem to be the compute cost of an output, but this is nowhere discussed or measured.
  • I'm unsure why the 7 domains chosen by the authors were selected over others as they are not broadly related to each other. I am also unsure these are a comprehensive enough set
  • There is no evidence the benchmark problems are high quality.
Questions for authors

See above.

2 comments

Thank you for the review. Section, figure and table numbers refer to the revision; results computed after submission are marked "new in the revision". W1. "heavy jargon which is never clearly introduced" We rewrote the introduction for readers outside the subfield. It now states the problem (software acting on model probabilities that pass format checks but carry no information), the gap and the main finding before any detail, and every term is defined where it first appears. The quoted paragraph is removed, and protocol bookkeeping moved to Appendix A (Sections 1, 3 and 5). W2. "The paper contains no figures." The revision has three figures: Figure 1 (concept: answer space, probability vector, outcome scoring), Figure 2 (main result: per-family NLL relative to uniform for every implementation) and Figure 3 (elicitation control across prompts and model sizes). W3. "All evaluations are on Qwen3-0.6B ... no study across scales" The revision now evaluates Qwen3 checkpoints at 0.6B, 1.7B, 4B and 14B parameters, plus supervised decision heads and up to four domain controls per family, on the same outcomes. The submitted results tables covered validity (submitted Table 4) and hosted-model scores (submitted Table 5); the revision reports outcome scores for all implementations (Section 6.1, Figure 2, Tables 2-3). New in the revision, an elicitation control runs Qwen3-4B and Qwen3-14B under the original prompt and an exemplar-free prompt on 1,251 states of six families (Section 6.2, Figure 3, Appendix B); the 1.7B rationale-conditioned readout is in Appendix L. Results on the test split: Copying of the prompt's format example persists at 4B and shrinks at 14B (0.89 and 0.27 of valid anchored outputs). Without the example, 52% (4B) and 29% (14B) of valid outputs are still integer-uniform. In five of six families a learned head beats every Qwen readout, raw and after calibration; larger is not uniformly better (highD one-pass NLL 0.964 at 4B, 1.931 at 14B). All checkpoints are from one model family, which Section 7 states as a limitation. W4. "why these tasks ... rather than many other benchmarks with a fixed output format" / cost Here the answer space is built per decision from the state known at the cutoff, so the number of events, their ordinal or categorical type and the partition can change between examples. Predictions are scored against outcomes recorded after a horizon, with invalid outputs kept in the denominator and intervals over dependent groups of examples. Fixed-format benchmarks score a fixed label set against human-provided labels. We did not run a fixed-label comparison arm, so we do not claim a measured difference from such benchmarks (Sections 1 and 3). On cost, a factual correction: the submission did discuss it. The abstract, the introduction and Section 3 named it, and Appendix D ("Cost measurement protocol") specified local latency, throughput and package-energy measurement. You are right that no cost number was produced: the timed runs aborted under background contention before timed inference. The revision states once that cost is unmeasured and that isolated-host measurement is future work (Section 7, Appendix J). W5. "why the 7 domains ... were selected" The families differ in answer-space size and type, in horizon (one second to one day) and in dependence structure (sessions, trading days, collector days, recordings, dates, vessel-dates, agent components), and each has an outcome log that resolves after the horizon. They span six domains. We do not claim the set is comprehensive (Section 4, "Why these seven families"). W6. "no evidence the benchmark problems are high quality" The revision reports the existing checks in Section 4, "Label and source quality": LOB: 18 of 65 sessions pass integrity checks, giving 21,402 eligible outcomes. Expanded mempool: 47 of 75 source files are excluded by source checks. Weather: the Daymet population audit finds 0 censored windows. AIS: all 4,608 retained attempts have valid outcome labels. Agent: a census of 164 tasks and 2,624 episodes; 48 episodes with defective references are excluded, and labels come from pinned replay. highD: held-out recordings were audited for split separation. Labels are recorded outcomes mapped to events by a deterministic rule, and answer spaces are built before outcomes are attached (Section 3).

I will maintain my score.

Reviewer_nV7JOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SystemOneBench specifies evaluation of typed probability distributions against outcome-derived labels across seven data families. It defines temporal and entity splits, proper scoring, calibration, selective prediction, option perturbations, and cost measurement. Experiments include three readouts of Qwen3-0.6B and a hosted decision service. Free and constrained generation copy a uniform format example on all 15,491 outputs per readout. The hosted service is worse than uniform on agent-trace log loss and assigns zero probability to observed calibration events in AIS and weather, preventing finite-loss temperature fitting.

Critical strengths
  1. The probability contract addresses important evaluation failures: invalid attempts remain in coverage denominators, exact zeros remain visible, and answer spaces are constructed independently of subsequent outcomes. Declared dependence units also prevent treating overlapping observations as independent evidence.

  2. The reported diagnostics make the distinction between format validity and predictive information concrete. Template copying and zero-probability errors are consequential failures for software that consumes model probabilities directly.

Critical weaknesses
  1. Most of the proposed benchmark’s empirical validation is absent. Section 8 explicitly leaves the family-wide proper-score, calibration, selective-risk, shift, option-perturbation, and cost analyses unreported. The available results therefore do not establish how useful local readouts, learned heads, and domain controls perform across the benchmark.

  2. The local generative comparison uses one 0.6B checkpoint and a prompt known during the formatting pilot to induce copying. This establishes an elicitation failure case, but offers limited evidence about the benchmark’s ability to distinguish functioning probabilistic decision systems. The manuscript appropriately limits architecture claims, yet the empirical contribution remains narrow.

Questions for authors
  1. Assessment-relevant evidence: Can you report the primary paired proper-score results from the saved predictions, including one-pass scoring, learned heads, domain controls, and uniform references, with the declared dependence-aware intervals? These results are needed to assess whether the benchmark measures useful decision quality beyond the reported failure cases.

  2. Elicitation control: Zhao et al., “Calibrate Before Use: Improving Few-Shot Performance of Language Models” (ICML 2021), not cited in the manuscript, documents substantial prompt-induced answer bias. Does template copying persist with alternative format examples or another functioning probability-elicitation prompt, and how do outcome scores change? This would distinguish a prompt-specific failure from a stable limitation of the evaluated readout.

https://proceedings.mlr.press/v139/zhao21c.html

1 comment

Thank you for the review and for the Zhao et al. pointer. Section, figure and table numbers refer to the revision; results computed after submission are marked "new in the revision". W1/Q1. "Most of the proposed benchmark's empirical validation is absent" / "primary paired proper-score results" The revision reports them for all seven families (new in the revision). They were specified in the submitted protocol and computed from the saved predictions with the protocol's code, but the submission did not report them (submitted Section 8, "Scope of the reported results"). The revision adds: per-family NLL and Brier score of one-pass scoring, learned heads (three seeds), the tuning-selected domain control, the hosted model and the uniform reference (Section 6.1, Figure 2, Table 2); the pre-specified paired comparisons with dependence-aware 95% intervals from 1,000 resamples of whole dependence units (Table 3; all partitions in Appendices N-O); temperature calibration (Section 6.3, Appendices F and P), selective prediction with frozen calibration thresholds against random abstention (Appendix S), option permutation and paraphrase diagnostics with held-out answer spaces and strata (Appendices F and Q), and an expanded multi-date mempool cohort (Section 6.3, Appendix C). Main reading: a learned head or the selected domain control has the lowest NLL in five of seven families; one-pass scoring exceeds uniform in 14 of the 18 original test partitions; the hosted model is lowest on weather; on Agent nothing beats uniform. On Agent, the hosted model minus uniform is +0.105 [-0.029, 0.239] in raw NLL over 11 owner components, so its disadvantage is not established (Section 6.3). Only AIS and Agent reach ten test dependence units; the other families are reported as descriptive of their observed units (Table 1). W2/Q2. "one 0.6B checkpoint and a prompt known ... to induce copying" / elicitation control Copying is prompt- and model-dependent, but removing it does not produce useful probabilities from these checkpoints. The revision adds an elicitation control and cites Zhao et al. (2021) on prompt-induced answer bias (Section 2; new in the revision; Section 6.2, Figure 3, Appendix B). Why the anchored prompt was kept: in the twelve-input formatting pilot, the plain prompt exhausted its output budget on all twelve tuning inputs, and only the example prompt produced parseable arrays. Prompts were fixed before test scoring, so we reported the copying as a measured failure. Setup: the same 1,251 states of six families (675 test states; mempool is not in this cohort), with Qwen3-4B and Qwen3-14B, each running one-pass scoring and free and constrained generation under the original prompt with its uniform format example and under an exemplar-free prompt that has no numerical example and asks for larger weights on more likely options. Results on the test split: Copying persists at 4B and shrinks at 14B: 0.89 (1,200 of 1,349) and 0.27 (370 of 1,348) of valid anchored outputs equal the example, against all attempts at 0.6B. This share is an upper bound on template copying, since a model can return uniform weights without any example. Without the example, outputs vary with the input more often, yet 52% (4B) and 29% (14B) of valid outputs are still integer-uniform, and in IEX, weather and AIS several generative readouts put exactly zero probability on realized test outcomes (infinite NLL). Outcome scores: in five of six families a learned head beats every Qwen readout, raw and after temperature calibration. On Agent no raw Qwen readout beats uniform (best 0.725 vs 0.693). Larger is not uniformly better: highD one-pass NLL is 0.964 at 4B and 1.931 at 14B. Outcome scoring is what reveals that the exemplar-free outputs, although less copied, are still not useful probabilities. Checkpoint, precision and device change together relative to the 0.6B CPU runs. We did not run a skewed (non-uniform) format example; Section 7 notes it as future work. The study still covers one model family, Qwen3 from 0.6B to 14B, which Section 7 states as a limitation.

Reviewer_6iQzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SystemOneBench proposes a benchmark for models that return probability distributions over explicitly defined answer spaces. It covers seven data families across six domains, with labels derived from observed outcomes. The protocol specifies evaluation of probability quality, calibration, selective prediction, robustness, and computational cost. Initial experiments examine three readouts of Qwen3-0.6B and the hosted Jev model, showing that valid outputs can still be uninformative or assign zero probability to events that occur.

Critical strengths
  • The paper addresses a useful evaluation problem. Scoring predictions against observed outcomes provides a direct test of reliability that cannot be established by schema validity or agreement with model-generated reference probabilities.

  • The shared interface across heterogeneous domains is potentially valuable. Explicit event partitions and consistent probability requirements provide a common basis for comparing generative readouts, direct option scoring, and supervised decision models.

  • The protocol pays attention to details that are easy to overlook, including temporal cutoffs, dependence between examples, invalid-attempt accounting, and exact-zero probabilities. The authors are also transparent about source limitations and the restricted scope of the reported findings.

Critical weaknesses
  • The empirical evaluation is incomplete for a benchmark paper. Section 8 explicitly states that most proper-score, calibration, selective-risk, shift, option-perturbation, and cost comparisons are not reported. Without these results, it is difficult to assess whether the benchmark meaningfully distinguishes useful decision models.

  • Both generative baselines copy the uniform format example on every input, and this behavior was already observed during the formatting pilot. This is a useful failure case, but it leaves the paper without a functioning generative baseline and provides limited insight into the relative merits of different readouts.

  • The number of independent evaluation units is small in several families, despite the larger example counts. Some temporal tests cover only one or two days, and the reported Agent comparison lacks dependence-aware uncertainty estimates. This limits confidence in conclusions about performance under distribution shift.

Questions for authors
  1. Can the authors provide the central per-family outcome scores for one-pass scoring, learned heads, and domain controls, together with the calibration and quality–cost comparisons specified in the protocol?

  2. Why was the example prompt retained after the pilot showed complete copying? Does this behavior persist with stronger instruction-tuned models or alternative prompts that avoid a copyable probability vector?

  3. How stable is the hosted model’s disadvantage to the uniform baseline across the eleven Agent test components? Please provide the paired uncertainty estimate and clarify which other evaluation splits support conclusions beyond their sampled periods.

1 comment

Thank you for the careful review. Section, figure and table numbers refer to the revision; results computed after submission are marked "new in the revision". W1/Q1. "The empirical evaluation is incomplete" / "central per-family outcome scores" The revision reports these results for all seven families (new in the revision). The submitted protocol specified them, and they were computed from the saved predictions with the protocol's code, but the submission did not report them (submitted Section 8, "Scope of the reported results"). The revision adds: per-family NLL and Brier score of one-pass scoring, learned heads (three seeds), the tuning-selected domain control, the hosted model and the uniform reference (Section 6.1, Figure 2, Table 2); the pre-specified paired comparisons with 95% intervals from 1,000 resamples of whole dependence units (Table 3; all partitions in Appendices N-O); temperature calibration (Section 6.3, Appendices F and P), selective prediction with frozen calibration thresholds, reporting coverage and risk against random abstention (Appendix S), option permutation and paraphrase diagnostics with held-out answer spaces and strata (Appendices F and Q), and an expanded multi-date mempool cohort (Section 6.3, Appendix C). Main reading: a learned head or the selected domain control has the lowest NLL in five of seven families; one-pass scoring exceeds uniform in 14 of the 18 original test partitions; the hosted model is lowest on weather; on Agent nothing beats uniform. The benchmark therefore separates implementations that are all schema-valid. The quality-cost comparison is not available. The protocol specified it (submitted Appendix D), but the timed runs aborted under background contention before any timed inference, so no cost number exists. The revision states this once (Section 7, Appendix J). W2/Q2. "copy the uniform format example" / "Why was the example prompt retained" In the twelve-input formatting pilot, the plain prompt exhausted its output budget on all twelve tuning inputs, and only the example prompt produced parseable arrays. Prompts were fixed before test scoring, so we reported the copying as a measured failure instead of tuning it away on test data (Section 6.2; Appendix A). To test whether copying is prompt-specific, the revision adds an elicitation control (new in the revision; Section 6.2, Figure 3, Appendix B). On the same 1,251 states of six families (675 test states; mempool is not in this cohort), Qwen3-4B and Qwen3-14B each run one-pass scoring and free and constrained generation under the original prompt and under an exemplar-free prompt that has no numerical example and asks for larger weights on more likely options. On the test split: Copying persists at 4B and shrinks at 14B: 0.89 (1,200 of 1,349) and 0.27 (370 of 1,348) of valid anchored outputs equal the example, against all attempts at 0.6B. This share is an upper bound on template copying, since a model can return uniform weights without any example. Without the example, outputs vary with the input more often, but 52% (4B) and 29% (14B) of valid outputs are still integer-uniform, and in IEX, weather and AIS several generative readouts put zero probability on a realized test outcome (infinite NLL). In five of six families a learned head beats every Qwen readout, raw and after temperature calibration. On Agent no raw Qwen readout beats uniform (best 0.725 vs 0.693). Larger is not uniformly better: highD one-pass NLL is 0.964 at 4B and 1.931 at 14B. So copying depends on prompt and checkpoint, but removing it does not yield useful probabilities from these checkpoints. Checkpoint, precision and device change together relative to the 0.6B CPU runs. We did not run a skewed (non-uniform) format example; Section 7 notes it as future work. W3/Q3. "number of independent evaluation units is small" / hosted model on Agent We agree. Intervals resample whole dependence units, and families below ten test units are labelled descriptive (Section 3, Table 1). For the hosted model minus uniform on the 11 Agent test owner components (new in the revision, Section 6.3): raw ΔNLL +0.105 [-0.029, 0.239], ΔBrier +0.096 [-0.031, 0.222]; after temperature calibration ΔNLL +0.082 [-0.033, 0.196]. All intervals include zero, so the submitted statement that the hosted model is worse than uniform on Agent is a point estimate whose sign is not established; the revision says so. Per-component values are not reported. Only AIS and Agent reach ten test dependence units. For LOB, IEX, mempool, highD and weather, results describe the observed sessions, days, recordings and dates (Table 1, Appendix G). Forward-time splits have no training-period reference, so they are evaluation levels, and we claim no identified shift effect (Section 7, Appendix A).

Comments on the paper

1 comment

We thank the three reviewers. The reviews agreed on four points: the submission reported too few outcome scores, the generative baseline copied its format example, the paper was hard to read and had no figures, and some families have few independent test units. The revision addresses each. Locations refer to the revision; "new in the revision" marks results computed after submission. Main finding. Nearly all outputs of the evaluated implementations are valid probability distributions, yet outcome scores separate them. A learned decision head or the tuning-selected domain control has the lowest NLL in five of seven families; Qwen3 readouts mostly stay near or above the uniform reference, copy the prompt's format example, or put zero probability on realized events (Section 6.1, Figure 2).

  1. Full outcome scores (6iQz, nV7J). New in the revision, for all seven families: NLL and Brier score of one-pass scoring, learned heads (three seeds), the selected domain control, the hosted model and the uniform reference (Figure 2, Table 2); the pre-specified paired comparisons with 95% intervals from 1,000 resamples of whole dependence units (Table 3); and calibration, selective-prediction, robustness and held-out diagnostics (Section 6.3). These were specified in the submitted protocol and computed from the saved predictions with its code, but not reported in the submission. One-pass scoring exceeds uniform in 14 of the 18 original test partitions; the hosted model is lowest on weather; on Agent nothing beats uniform.
  2. Elicitation control (6iQz, nV7J, kUD4). New in the revision (Section 6.2, Figure 3, Appendix B). On 1,251 states of six families (675 test states), Qwen3-4B and Qwen3-14B run one-pass scoring and free and constrained generation, under the original prompt (with its uniform format example) and an exemplar-free prompt. On the test split, copying persists at 4B (0.89 of valid anchored outputs) and shrinks at 14B (0.27). Without the example, 52% (4B) and 29% (14B) of valid outputs are still integer-uniform, and several generative readouts put zero probability on realized outcomes in IEX, weather and AIS. In five of six families a learned head beats every Qwen readout, raw and after calibration. Copying is prompt- and model-dependent, but removing it does not yield useful probabilities from these checkpoints.
  3. Uncertainty and the hosted model on Agent (6iQz). New in the revision, the hosted model minus uniform on the 11 Agent test owner components is +0.105 [-0.029, 0.239] in raw NLL and +0.082 [-0.033, 0.196] after calibration. The intervals include zero, so the submitted "worse than uniform" statement is a point estimate whose sign is not established (Section 6.3). Only AIS and Agent reach ten test dependence units; the other families are labelled descriptive of their observed units (Table 1, Section 7).
  4. Writing, figures, scope and design (kUD4, nV7J). Rewritten introduction, three new figures, and the rationale for the seven families with their label and source checks: see the summary of changes below. All Qwen3 checkpoints come from one model family.
  5. Cost (kUD4, 6iQz). The submission specified a cost measurement protocol (submitted Appendix D), but the timed runs aborted under background contention before timed inference, so no cost number exists. The revision states this once (Section 7, Appendix J); we report no new cost results.

Summary of changes (revision round 1) "New" marks results computed or text added after submission; reviewers are in brackets. NLL and Brier score for all seven families (new): Section 6.1, Figure 2, Table 2 [6iQz, nV7J, kUD4]. Pre-specified paired comparisons with dependence-unit intervals (new): Table 3, Appendices N-O [6iQz, nV7J]. Calibration, selective prediction, option permutation and paraphrase, held-out answer spaces and strata, expanded mempool cohort (new): Section 6.3, Appendices C, F, P, Q, S [6iQz, nV7J]. Elicitation control (new) and why the anchored prompt was kept: Section 6.2, Figure 3, Appendices A, B [6iQz, nV7J, kUD4]; Zhao et al. (2021) cited in Section 2 [nV7J]. Hosted model minus uniform on Agent with intervals (new); support floor and descriptive labels for families below ten test units: Sections 3, 6.3, 7, Tables 1, 3, Appendices A, G [6iQz]. Introduction rewritten, terms defined at first use, bookkeeping moved to Appendix A; three figures added: Sections 1, 3, 5, Figures 1-3 [kUD4]. Why these seven families, and label and source quality: Section 4 [kUD4]. Cost stated once as unmeasured: Section 7, Appendix J [kUD4, 6iQz]. Model scope (Qwen3 0.6B, 1.7B, 4B, 14B; one model family): Sections 5, 7 [kUD4, nV7J]. Not done: quality-cost comparison (no cost measurement exists); skewed format example (future work, Section 7); per-component Agent values; a fixed-label benchmark comparison arm.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.