acceptodds
Under review as a conference paper at ICLR 2027

Honkaku-Bench: Can AI solve fair-play murder mysteries?

Abstract

Evaluating whether a large language model can reason, rather than rely solely on retrieval or pattern matching, requires assessing not only its final answer but also the validity of the reasoning steps supporting it. Honkaku mysteries offer a natural setting for this evaluation as fair-play puzzles challenge readers to explain seemingly impossible events using clues in the story. We introduce Honkaku- Bench, a benchmark of 70 mystery puzzles for evaluating evidence-grounded reconstruction, paired with unique official solutions written by human experts. Given a complete case file and any accompanying diagrams, models must produce an open-ended solution explaining what happened, how it was possible, and which evidence supports the reconstruction. Across eight reasoning models evaluated through a common harness, GPT-5.6-sol and Fable 5 recover 61.3% and 59.3% of rubric credit on average but receive full credit on only 6 and 8 of 70 cases. All eight models score 18.1–26.8 points lower on the cases with visual evidence than on the text-only cases. Our case studies reveal how partial progress fails to build a coherent reconstruction: a model may recover the mechanism while naming the wrong culprit, or propagate one misread clue through consistent arguments. Honkaku-Bench thus tests whether models can turn scattered textual and visual clues into one explanation consistent with the entire case.

Primary area:
datasets and benchmarks
Keywords:
benchmark, murder mystery, reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zegnOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HonkakuBench, a benchmark of 70 human-authored fair-play mystery cases for evaluating complete, evidence-grounded reconstructions. Each case includes an official solution and an author-verified scoring rubric; 46 cases contain visual evidence. Models receive the complete case materials and produce an open-ended solution, which is graded at the criterion level by an LLM judge. Across eight evaluated models, the two leading models obtain average rubric scores of 61.3% and 59.3%, while receiving full credit on only 6 and 8 cases, respectively. The paper also reports analyses across evidence modalities, reasoning-demand categories, languages, execution configurations, and grading models, and attributes many failures to difficulties reconciling locally correct components into a globally consistent explanation.

Critical strengths
  1. The task is meaningfully motivated. Requiring a complete reconstruction can reveal distinctions that culprit-only accuracy would miss.
  2. Official solutions and rubrics are supplied by the case authors, who subsequently verify the restructured criteria and weights (§3.3). The inclusion of visual evidence in 46 cases also provides potentially useful material for studying evidence integration.
  3. The paper makes several efforts to examine evaluation reliability. These include grading identical submissions with three judges, a blinded human calibration study, and comparisons across execution configurations and languages. The limitations section appropriately acknowledges several confounds, including the lack of paired text-only versions of visual cases and the effects of configuration changes.
Critical weaknesses
  1. The benchmark’s coverage and robustness are insufficiently established. The dataset contains only 70 cases from one competition, with subgroup analyses based on even smaller subsets. The paper does not adequately justify their coverage of the intended task or report case-level uncertainty estimates. With one saved submission per model–case pair, the evidence is insufficient to establish the stability and generalizability expected of a benchmark for comparing reasoning capabilities.
  2. The validity and reliability of the scoring system require stronger evidence. Although the case-specific rubrics are author-verified, the principles governing criterion decomposition, weights, and partial credit remain underspecified. Normalization does not establish comparable measurement across cases, and sensitivity to alternative scoring choices is unexplored. Human calibration covers only four cases, with judge–human mean absolute differences of approximately 11–12 points (Table 13). These limitations weaken confidence in the interpretation of aggregate scores and completion rates.
Questions for authors
  1. Dataset coverage and stability: How were the 70 cases selected from the candidate pool, and what supports their coverage of the intended reasoning tasks? Can case-level resampling or paired uncertainty estimates establish the stability of the principal model and subgroup comparisons?
  2. Rubric design: What common principles determine criterion decomposition, weights, partial credit, and the treatment of dependent or contradictory claims? Would the main conclusions remain unchanged under reasonable alternative weighting or scoring choices?
  3. Grading reliability: How were the four cases used for human calibration selected, and how representative are they? Given the approximately 11–12-point judge–human errors and the 6.3–16.3-point shifts under alternative judges, what evidence supports the reliability of the 90% and 100% completion thresholds?
  4. Capability-level interpretation: What evidence distinguishes a specific “global reconciliation” bottleneck from the ordinary difficulty of satisfying all components of a complex task? Beyond descriptive comparisons and selected cases, is there a controlled analysis that separates evidence acquisition, local inference, and global consistency failures?
1 comment

We genuinely appreciate the reviewer for the comprehensive comments concerning reasoning benchmarks for LLMs. Please see our point-to-point response to your comments below.

Coverage and Stability

We assessed sensitivity to case selection through resampling, obtaining 95% case-bootstrap intervals of about plus-minus 6 points on each model mean. GPT-5.6-sol and Fable 5 ranked above the other four models in all of the resamples. Independent repeated runs of Fable 5 and Opus 4.8 correlate at r = 0.77 and 0.80 across cases, with nearly unchanged means on completed runs. These results support the separation of the leading models.

Our 70 cases provide broad coverage across five reasoning demands: spatial (29 cases), temporal (41), physical (36), identity (37), and rule-based reasoning (33). These categories overlap, with many cases requiring several reasoning abilities to be integrated into one consistent reconstruction. The benchmark also includes 46 cases with visual evidence and 24 text-only cases, covering diverse evidence formats alongside reasoning demands.

Rubric Design

We reuse the competition’s criteria and weights, reviewed with the original authors to ensure coverage of critical reasoning checkpoints and appropriate emphasis on key findings such as culprit identity. Partial credit follows the official rubric, and contradictory claims receive no credit. Based on these discussions as well as with the authors of the cases, we do not expect significant changes to the main qualitative conclusions under reasonable alternative weights.

Grading Reliability

The initial four cases were randomly selected and included visual and text-only cases with diverse reasoning demands. We have since expanded human grading to all 70 cases, addressing the limited initial coverage. The judge–human mean absolute difference is 8 points, compared with 6.5 points between human graders.

The competition rubric and the 90%/100% cutoffs are held fixed across judges, although assigned scores remain judge-dependent. For qualitative analysis, the model judges’ overly generous awards at these thresholds are the main source of disagreement. Empirically, Opus 4.8 receives ≥\ge 90%/100% credit on 5/2 cases under the primary judge, versus 1/0 under the alternative GPT judge (Table 10). This supports the qualitative finding that complete reconstruction remains uncommon, while showing that exact completion counts are sensitive to grading. Although exact completion counts vary across judges, the persistent gap between partial credit and complete solutions supports HonkakuBench’s value in measuring progress from recovering individual solution components to reconstructing an entire mystery.

Capability-Level Interpretation

We use “global reconciliation” to describe a specific requirement within complex reasoning: the culprit, mechanism, timeline, and evidence must jointly satisfy the case constraints. HonkakuBench supplies the complete case upfront, removing the need for external retrieval or interactive evidence acquisition and focusing evaluation on processing and integrating available information.

Clue identification and local inference are still necessary, so this setup does not causally isolate a reconciliation-specific bottleneck. Our criterion-level case studies instead document observable mismatches, such as correct mechanisms paired with incorrect culprits. This makes HonkakuBench useful for measuring whether success on individual components translates into a coherent whole-case reconstruction, beyond what culprit accuracy alone captures.

Reviewer_enMROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces HonkakuBench, a benchmark of 70 Honkaku detective mysteries for evaluating multimodal evidence-grounded mystery reasoning in LLMs. Unlike existing benchmarks that primarily assess final-answer correctness, HonkakuBench requires models to reconstruct the culprit, mechanism, and supporting evidence through open-ended explanations, evaluated using a set of verified rubrics. The benchmark also includes 46 cases with visual evidence and supports three languages. Evaluations across frontier models show that the benchmark is challenging, with the strongest models achieve approximately 60% average rubric credit but could only fully solve 6-8 of the 70 cases. Performance is consistently lower on puzzles with visual evidence, and failure analyses suggest that models struggle to reconcile locally correct deductions into globally consistent explanations.

Critical strengths
  • the benchmarks are human written with author-verified scoring rubrics, and the addition of multimodal evidence is novel compared to prior mystery reasoning benchmarks. The rubrics also provide fine-grained evaluation metrics beyond mere final accuracy.
  • the authors evaluated a comprehrensive set of models, demonstrating that the benchmark is challenging for frontier models. The finding of 18.1–26.8 points decrease model performance among puzzles with visual evidence is also interesting.
Critical weaknesses
  • while the puzzles are human authored and seem high quality, 70 mysteries from a single writing competition could represent a relatively narrow distribution, and the authors provided limited analysis on the experts / puzzle selection process and puzzle difficulty.
  • the failure analysis, which heavily rely on LLM-generated labels, seems more descriptive than explanatory. Their central observation is that models can locally identify some correct clues or mechanisms without correctly reconstructing the entire mystery, but that is arguably not surprising -- any multi-step reasoning problem could have partially derived solutions. The authors have not demonstrated whether HonkakuBench has revealed novel failure modes or insights in AI reasoning
Questions for authors
  • could the authors provide more information on how the 70 mysteries were selected from the original competition, and provide some analysis on their perceived difficulty?
  • could the authors provide an interpretation on "All eight models score 18.1–26.8 points lower on the cases with visual evidence than on the text-only cases" -- is it indicative of models' multimodal perception failures, or puzzles with visual evidences are just more difficult in general?
  • in general I wish the authors could provide a more insightful error anlysis section. Some ideas (these are just suggestions and the authors don't have to follow them!) could be, is there a way to quantitatively measure how error propagates in the mystery solving process? Could model accuracy be improved by making them solve decomposed sub-problems first and then combining the intermediate results?
1 comment

We genuinely appreciate the reviewer for the comprehensive comments concerning reasoning benchmarks for LLMs. Please see our point-to-point response to your comments below.

On Visual Case Interpretation

We thank the reviewer for pointing out the result and agree that the 18.1–26.8-point gap compares different cases and does not isolate a visual effect. To better compare the reasoning differences when facing multimodal evidences, we ran a paired controlled experiment where we held each case fixed and varied only how its visual evidence is presented. An auditor model (Opus 4.8, given the official solution and rubric) found that 10 of the 46 diagram cases depend strongly on their images, 29 weakly, and 7 not at all. On all 10 strongly dependent cases, we ran GPT-5.6-sol twice under three conditions: (A) original images; (B) each image replaced in place by a text description, written without access to the solution and verified by a second model against the solution for completeness and absence of inference; and (C) images removed. Removing the images lowers the mean score by 8.0 points, so the images carry information the model uses. Replacing them with text descriptions does not raise scores, which gives no evidence of a perception bottleneck. With all visual information supplied as text, these cases still score 42.0, versus 72.8 on the 24 text-only cases under the same grader, and the 7 diagram cases whose images carry no needed information show a gap as large as the strongly visual ones (e.g., 40.6 vs. 22.8 points for GPT-5.6-sol). The gap therefore mainly reflects the intrinsic difficulty of the diagram cases rather than visual perception.

Selection and difficulty.

The 70 cases came from a private national mystery case solving competition, with targeted competitors all collegiate and above. The mystery cases cover spatial, temporal, physical, identity, and rule-based reasoning which demonstrate diversity within the benchmark. In the competition, the human solvers' mean scores range from 37.9% to 82.8% across 20 teams. We believe that these results along with the contrast of LLM's performance would contextualize difficulty demonstrate the expertise in the benchmark.

Failure Analysis Contribution

We agree with the reviewer that partial correctness alone is not a novel insight. However, we note that cases in HonkakuBench already include explicit subquestions that guide solving and support criterion-level grading, providing a natural decomposition. Moreover, our failure analysis in case studies identifies distinct reasoning breakdowns that outcome metrics would otherwise obscure in numbers.

For instance, in Case 06, GPT-5.6-sol recovers the mechanism but names the wrong culprit, whereas Opus 4.6 identifies the culprit while recovering substantially less of the mechanism. In Case 08, all eight models misinterpret the decisive clue, and several reason coherently from that incorrect premise. These cases distinguish failures to reconcile recovered components from failures to validate underlying premises, which were made possible due to the natural criterion-level design of HonkakuBench.

Lastly, we want to note that quantifying error propagation across long, interdependent reasoning chains is challenging because submitted answers provide incomplete reasoning traces, and their presentation order need not reflect the model’s reasoning order. Our contributions are therefore examine observable dependencies through criterion-level analysis and provide a diagnosis of whole-case reasoning that makes premise interpretation, mechanism recovery, and cross-component consistency separately assessable, providing concrete targets for evaluating reasoning improvements.

Reviewer_1BE1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper gives us HonkakuBench, 70 fair-play murder-mystery puzzles (46 with diagrams, 24 text-only, in ZH/EN/JA) for testing whether LLMs can reconstruct a whole case rather than just name an answer. Models get the full case file and must produce an open-ended solution — culprit, mechanism, motive, clues, reasoning — scored at criterion level against author-written rubrics. Eight models run through a shared harness, with Opus 4.8 as primary judge and a human calibration on top. The leaders (GPT-5.6-SOL, Fable 5) recover about 60% of rubric credit on average but fully solve only 6/70 and 8/70 cases. Every model drops 18–27 points on visual cases, and failures are mostly flawed deduction (62%) or wrong conclusion (27%), not missed clues.

Critical strengths
  • The judge is actually checked against humans — r = 0.89/0.90 with 74.3% exact criterion agreement (Table 13). That's what makes the ~60%-vs-6/70 gap believable instead of a grading artifact, and it's a step most LLM-judged benchmarks skip.
  • The "right pieces, wrong whole" finding. Case 06 nails the mechanism while naming the wrong culprit; Case 08 reasons coherently off a misread premise. A culprit-only or answer-only score would never surface this.
  • The rubrics are handled cleanly — author-written, restructured, re-verified by the author, then frozen across every run. No per-model rubric tuning, no judge drift.
  • The scope is genuinely multimodal and multilingual: 46 cases with 132 diagrams, three languages — broader than any prior mystery benchmark.
  • The paper probes its own weaknesses — alternative judges, harness ablation, cross-lingual, run-stability (Table 12) — and is honest about where it breaks.
Critical weaknesses
  • The "visual cases are harder" headline is confounded by case identity. The 46 visual and 24 text-only cases aren't paired (Section F says so), so the 18–27 point gap could just mean those cases are harder in their text. The abstract runs with it anyway.
  • The judge setup is a bit circular and thinly validated. Opus 4.8 grades the benchmark and is also one of the models being graded, same for the sensitivity judges. And the human check is 32 responses from 4 of 70 cases, with judge-human score error (~12) about double the human-human error (6.5).
  • Small corpus, single-run instability. 70 cases, and later runs drop 3–6 points (Table 12) — so the 2-point gap between the two leaders is inside the noise. Fine-grained rankings won't hold.
  • No decontamination check reported. The cases come from a competition; if the entries or solutions are public anywhere, the numbers are contaminated. I don't see any check for that.
  • No human solver baseline. The paper reports model scores (60% mean, 6/70 full) but never says how humans do on the same 70 cases, so I can't tell whether 60% is near-human or far below, or how much headroom the benchmark has. Watson & Holmes, the closest prior benchmark, includes a human comparison group; this one only uses humans to calibrate the judge.
Questions for authors
  • Did you run a decontamination check to make sure the 70 cases and their official solutions aren't in the evaluated models' training data (e.g., if the source competition entries are publicly available)?
  • Can you give a paired control isolating the visual-evidence effect — text-only and visual versions of the same cases — or some evidence the visual cases aren't just harder in text?
  • The failure labels (flawed deduction vs wrong conclusion) are judge-assigned, and Appendix E says some labels "remain model-generated." Can you report human agreement on the labels themselves, not just the scores?
1 comment

We genuinely appreciate the reviewer for the comprehensive comments concerning reasoning benchmarks for LLMs. Please see our point-to-point response to your comments below.

Confounding Results on Visual and Non-Visual Cases

We thank the reviewer for pointing out the result and agree that the 18.1–26.8-point gap compares different cases and does not isolate a visual effect. To better compare the reasoning differences when facing multimodal evidences, we ran a paired controlled experiment where we held each case fixed and varied only how its visual evidence is presented. An auditor model (Opus 4.8, given the official solution and rubric) found that 10 of the 46 diagram cases depend strongly on their images, 29 weakly, and 7 not at all. On all 10 strongly dependent cases, we ran GPT-5.6-sol twice under three conditions: (A) original images; (B) each image replaced in place by a text description, written without access to the solution and verified by a second model against the solution for completeness and absence of inference; and (C) images removed. Removing the images lowers the mean score by 8.0 points, so the images carry information the model uses. Replacing them with text descriptions does not raise scores, which gives no evidence of a perception bottleneck. With all visual information supplied as text, these cases still score 42.0, versus 72.8 on the 24 text-only cases under the same grader, and the 7 diagram cases whose images carry no needed information show a gap as large as the strongly visual ones (e.g., 40.6 vs. 22.8 points for GPT-5.6-sol). The gap therefore mainly reflects the intrinsic difficulty of the diagram cases rather than visual perception.

We will revise the abstract and update the numbers accordingly.

Judge Reliability

We have expanded human grading to cover all 70 benchmark cases. The judge–human mean absolute difference is 8 points, compared with 6.5 points between human graders. The main source of disagreement is model judges awarding 100% credit too generously.

Contamination Check

For decontamination check, we conducted an online extensive check and found no prior public copies of the competition materials. The original authors also confirmed that they had not uploaded these materials online before our benchmark release.

Failure Labels

Our failure-labeling procedure follows the same general framework as MAST [1] which is taxonomy-based LLM annotation combined with human validation. The judge compares each response with the official solution and rubric to assign a dominant failure label. A collaborator checked labels in the original seven-model set; we will complete the remaining checks and report human–human and judge–human label agreement separately from score agreement.

[1] Mert Cemri, Melissa Z. Pan, Shuyi Yang, et al. Why Do Multi-Agent LLM Systems Fail? NeurIPS 2025 Datasets and Benchmarks Track. arXiv:2503.13657.

Human Solver Baseline

We thank the reviewer for this suggestion. We obtained the official scores in the original competition for 20 teams on seven of the benchmark cases (cases 01–07). We report these as a human baseline on these seven cases, normalized to percentages and counting a missing submission as 0, as we do for models.

Solver (7 cases)Mean≥90%100%
Human teams, mean60.743/14018/140
Fable 575.14/74/7
GPT-5.6-sol72.73/72/7
Opus 4.864.23/72/7
GPT-5.561.92/70/7
Opus 4.655.32/71/7
Sonnet 530.60/70/7

The strongest models score above the median team (ranking 7th and 8th among the 20 teams), but remain 12–14 points below the top-5 teams and 21–23 points below the best team. Every case was fully solved by at least two teams, so complete reconstructions are achievable by humans on all seven cases. Human and model difficulty orderings largely agree (Spearman ρ=\rho= 0.89 between per-case human and mean model scores); the hardest case for both is case 07. We will add this comparison to the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.