- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper presents a benchmark for calibration in "System One" models, for example like the recent release of Jev, a model by TypeSafe AI. These models provide typed outputs that are constraint to fit in a certain format. This paper presents a benchmark for the evaluation of calibration in these models. The benchmarks contains tasks across a set of 7 data families. There is some evaluation on Jev and Qwen3-0.6B.
Critical strengths
- Thorough evaluation of these "System One" models to constrained decoding in LLMs is an important endeavor to understand their utility.
- It does seem important to have benchmarks for these models.
Critical weaknesses
- The writing in this paper is not easily comprehensible by humans. There is heavy jargon which is never clearly introduced and the as the reader it often feels that I am lacking context. The problem is not clearly motivated and the methods are not clearly explained. For example, without any background about the setup or details, this is a paragraph in the introduction: "The gap is not hypothetical. We elicited probability vectors from Qwen3-0.6B by free generation and by constrained generation that admits only valid vectors. Without a format example in the prompt, generation exhausted its output budget on every tuning input of a formatting pilot; with one, it completed"
- The paper contains no figures.
- The evaluation scheme is unclear and not comprehensive. All evaluations are on Qwen3-0.6B, which is too small to draw reasonable conclusions from and Jev, a closed source model without a public implementation. The only evaluation I can see is in Table 5, which is insufficient. There is no study across scales/models/architectures etc.
- There is also an issue in that the tasks here are not fundamentally different to many tasks that are solved by current LLMs. It's not clear why these tasks should be used to evaluate "System One" models rather than many other benchmarks with a fixed output format. The key difference between these models and LLMs seem to be the compute cost of an output, but this is nowhere discussed or measured.
- I'm unsure why the 7 domains chosen by the authors were selected over others as they are not broadly related to each other. I am also unsure these are a comprehensive enough set
- There is no evidence the benchmark problems are high quality.
Questions for authors
See above.
2 comments
I will maintain my score.
Thank you for the review. Section, figure and table numbers refer to the revision; results computed after submission are marked "new in the revision". W1. "heavy jargon which is never clearly introduced" We rewrote the introduction for readers outside the subfield. It now states the problem (software acting on model probabilities that pass format checks but carry no information), the gap and the main finding before any detail, and every term is defined where it first appears. The quoted paragraph is removed, and protocol bookkeeping moved to Appendix A (Sections 1, 3 and 5). W2. "The paper contains no figures." The revision has three figures: Figure 1 (concept: answer space, probability vector, outcome scoring), Figure 2 (main result: per-family NLL relative to uniform for every implementation) and Figure 3 (elicitation control across prompts and model sizes). W3. "All evaluations are on Qwen3-0.6B ... no study across scales" The revision now evaluates Qwen3 checkpoints at 0.6B, 1.7B, 4B and 14B parameters, plus supervised decision heads and up to four domain controls per family, on the same outcomes. The submitted results tables covered validity (submitted Table 4) and hosted-model scores (submitted Table 5); the revision reports outcome scores for all implementations (Section 6.1, Figure 2, Tables 2-3). New in the revision, an elicitation control runs Qwen3-4B and Qwen3-14B under the original prompt and an exemplar-free prompt on 1,251 states of six families (Section 6.2, Figure 3, Appendix B); the 1.7B rationale-conditioned readout is in Appendix L. Results on the test split: Copying of the prompt's format example persists at 4B and shrinks at 14B (0.89 and 0.27 of valid anchored outputs). Without the example, 52% (4B) and 29% (14B) of valid outputs are still integer-uniform. In five of six families a learned head beats every Qwen readout, raw and after calibration; larger is not uniformly better (highD one-pass NLL 0.964 at 4B, 1.931 at 14B). All checkpoints are from one model family, which Section 7 states as a limitation. W4. "why these tasks ... rather than many other benchmarks with a fixed output format" / cost Here the answer space is built per decision from the state known at the cutoff, so the number of events, their ordinal or categorical type and the partition can change between examples. Predictions are scored against outcomes recorded after a horizon, with invalid outputs kept in the denominator and intervals over dependent groups of examples. Fixed-format benchmarks score a fixed label set against human-provided labels. We did not run a fixed-label comparison arm, so we do not claim a measured difference from such benchmarks (Sections 1 and 3). On cost, a factual correction: the submission did discuss it. The abstract, the introduction and Section 3 named it, and Appendix D ("Cost measurement protocol") specified local latency, throughput and package-energy measurement. You are right that no cost number was produced: the timed runs aborted under background contention before timed inference. The revision states once that cost is unmeasured and that isolated-host measurement is future work (Section 7, Appendix J). W5. "why the 7 domains ... were selected" The families differ in answer-space size and type, in horizon (one second to one day) and in dependence structure (sessions, trading days, collector days, recordings, dates, vessel-dates, agent components), and each has an outcome log that resolves after the horizon. They span six domains. We do not claim the set is comprehensive (Section 4, "Why these seven families"). W6. "no evidence the benchmark problems are high quality" The revision reports the existing checks in Section 4, "Label and source quality": LOB: 18 of 65 sessions pass integrity checks, giving 21,402 eligible outcomes. Expanded mempool: 47 of 75 source files are excluded by source checks. Weather: the Daymet population audit finds 0 censored windows. AIS: all 4,608 retained attempts have valid outcome labels. Agent: a census of 164 tasks and 2,624 episodes; 48 episodes with defective references are excluded, and labels come from pinned replay. highD: held-out recordings were audited for split separation. Labels are recorded outcomes mapped to events by a deterministic rule, and answer spaces are built before outcomes are attached (Section 3).