- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces HonkakuBench, a benchmark of 70 human-authored fair-play mystery cases for evaluating complete, evidence-grounded reconstructions. Each case includes an official solution and an author-verified scoring rubric; 46 cases contain visual evidence. Models receive the complete case materials and produce an open-ended solution, which is graded at the criterion level by an LLM judge. Across eight evaluated models, the two leading models obtain average rubric scores of 61.3% and 59.3%, while receiving full credit on only 6 and 8 cases, respectively. The paper also reports analyses across evidence modalities, reasoning-demand categories, languages, execution configurations, and grading models, and attributes many failures to difficulties reconciling locally correct components into a globally consistent explanation.
Critical strengths
- The task is meaningfully motivated. Requiring a complete reconstruction can reveal distinctions that culprit-only accuracy would miss.
- Official solutions and rubrics are supplied by the case authors, who subsequently verify the restructured criteria and weights (§3.3). The inclusion of visual evidence in 46 cases also provides potentially useful material for studying evidence integration.
- The paper makes several efforts to examine evaluation reliability. These include grading identical submissions with three judges, a blinded human calibration study, and comparisons across execution configurations and languages. The limitations section appropriately acknowledges several confounds, including the lack of paired text-only versions of visual cases and the effects of configuration changes.
Critical weaknesses
- The benchmark’s coverage and robustness are insufficiently established. The dataset contains only 70 cases from one competition, with subgroup analyses based on even smaller subsets. The paper does not adequately justify their coverage of the intended task or report case-level uncertainty estimates. With one saved submission per model–case pair, the evidence is insufficient to establish the stability and generalizability expected of a benchmark for comparing reasoning capabilities.
- The validity and reliability of the scoring system require stronger evidence. Although the case-specific rubrics are author-verified, the principles governing criterion decomposition, weights, and partial credit remain underspecified. Normalization does not establish comparable measurement across cases, and sensitivity to alternative scoring choices is unexplored. Human calibration covers only four cases, with judge–human mean absolute differences of approximately 11–12 points (Table 13). These limitations weaken confidence in the interpretation of aggregate scores and completion rates.
Questions for authors
- Dataset coverage and stability: How were the 70 cases selected from the candidate pool, and what supports their coverage of the intended reasoning tasks? Can case-level resampling or paired uncertainty estimates establish the stability of the principal model and subgroup comparisons?
- Rubric design: What common principles determine criterion decomposition, weights, partial credit, and the treatment of dependent or contradictory claims? Would the main conclusions remain unchanged under reasonable alternative weighting or scoring choices?
- Grading reliability: How were the four cases used for human calibration selected, and how representative are they? Given the approximately 11–12-point judge–human errors and the 6.3–16.3-point shifts under alternative judges, what evidence supports the reliability of the 90% and 100% completion thresholds?
- Capability-level interpretation: What evidence distinguishes a specific “global reconciliation” bottleneck from the ordinary difficulty of satisfying all components of a complex task? Beyond descriptive comparisons and selected cases, is there a controlled analysis that separates evidence acquisition, local inference, and global consistency failures?
We genuinely appreciate the reviewer for the comprehensive comments concerning reasoning benchmarks for LLMs. Please see our point-to-point response to your comments below.
Coverage and Stability
We assessed sensitivity to case selection through resampling, obtaining 95% case-bootstrap intervals of about plus-minus 6 points on each model mean. GPT-5.6-sol and Fable 5 ranked above the other four models in all of the resamples. Independent repeated runs of Fable 5 and Opus 4.8 correlate at r = 0.77 and 0.80 across cases, with nearly unchanged means on completed runs. These results support the separation of the leading models.
Our 70 cases provide broad coverage across five reasoning demands: spatial (29 cases), temporal (41), physical (36), identity (37), and rule-based reasoning (33). These categories overlap, with many cases requiring several reasoning abilities to be integrated into one consistent reconstruction. The benchmark also includes 46 cases with visual evidence and 24 text-only cases, covering diverse evidence formats alongside reasoning demands.
Rubric Design
We reuse the competition’s criteria and weights, reviewed with the original authors to ensure coverage of critical reasoning checkpoints and appropriate emphasis on key findings such as culprit identity. Partial credit follows the official rubric, and contradictory claims receive no credit. Based on these discussions as well as with the authors of the cases, we do not expect significant changes to the main qualitative conclusions under reasonable alternative weights.
Grading Reliability
The initial four cases were randomly selected and included visual and text-only cases with diverse reasoning demands. We have since expanded human grading to all 70 cases, addressing the limited initial coverage. The judge–human mean absolute difference is 8 points, compared with 6.5 points between human graders.
The competition rubric and the 90%/100% cutoffs are held fixed across judges, although assigned scores remain judge-dependent. For qualitative analysis, the model judges’ overly generous awards at these thresholds are the main source of disagreement. Empirically, Opus 4.8 receives ≥ 90%/100% credit on 5/2 cases under the primary judge, versus 1/0 under the alternative GPT judge (Table 10). This supports the qualitative finding that complete reconstruction remains uncommon, while showing that exact completion counts are sensitive to grading. Although exact completion counts vary across judges, the persistent gap between partial credit and complete solutions supports HonkakuBench’s value in measuring progress from recovering individual solution components to reconstructing an entire mystery.
Capability-Level Interpretation
We use “global reconciliation” to describe a specific requirement within complex reasoning: the culprit, mechanism, timeline, and evidence must jointly satisfy the case constraints. HonkakuBench supplies the complete case upfront, removing the need for external retrieval or interactive evidence acquisition and focusing evaluation on processing and integrating available information.
Clue identification and local inference are still necessary, so this setup does not causally isolate a reconciliation-specific bottleneck. Our criterion-level case studies instead document observable mismatches, such as correct mechanisms paired with incorrect culprits. This makes HonkakuBench useful for measuring whether success on individual components translates into a coherent whole-case reconstruction, beyond what culprit accuracy alone captures.