- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether a model-independent structural difficulty measure predicts LLM-agent success in a small deterministic autobattler. Each instance has 3,240 ordered complete plays; their winning fraction d is also exactly the win probability of uniform random play. Two headline agents and a separately reported third model play 126 instances with ten attempts each, sampled from two separated difficulty ranges. A preregistered common-slope GLMM and four Holm-adjusted tests assess association with difficulty, levels at d=0.1, and overdispersion beyond a simpler difficulty-only model. Twenty-one scripted strategies illustrate that slope is not a capability score. A smaller prompt-intervention experiment reports regret changes when rules and exact move values are supplied.
Critical strengths
-
The structural measure has an exact and interpretable random baseline. Equal prices, fixed play length, and exhaustive enumeration make P(random win)=d a genuine property of the environment rather than a difficulty label inferred from the models under test. This is a useful controlled setup for studying what difficulty grading can and cannot tell us.
-
The scripted-policy comparisons make an important distinction between the level of success and its slope with difficulty. The paper also reports substantial limitations rather than hiding them: agent-specific fits, fallbacks, interpolation at an unobserved d, imperfect interval calibration, and the separate intervention arm's unresolved gate verdict. This transparency makes the study easier to assess and its narrower descriptive findings potentially useful.
Critical weaknesses
-
The central heterogeneity interpretation is not separated from misspecification of the null model. Appendix B.5 defines Q4 as overdispersion against a binomial GLM with model intercepts and one common logit(d) slope. Yet Table 2 reports different agent-specific slopes (1.4429 versus 1.0079), and Appendix B.8 reports different instance-effect SDs (2.3307 versus 1.0889). Rejecting this restricted GLM can reflect agent-specific slopes or an incorrectly specified response curve, as well as residual instance difficulty. It does not by itself isolate differences between instances at equal d. The same issue weakens the pooled claim that the two agents respond about as steeply as random play. A heterogeneity analysis allowing model-specific response functions, or a direct comparison of genuinely matched-d instances, is needed for the title-level conclusion.
-
Important inferential claims remain dependent on unverified structure. Both level tests use d=0.1, but all sampled instances lie in 0.0164–0.0265 or 0.1735–0.3762, so the estimates depend on an untested interpolation. The model-free suite-average advantage in Appendix B.8 supports an advantage on observed instances, not a rate at 0.1. Separately, Section 7's regret attribution cannot distinguish search from valuation and retains a harness/provider-mismatch verdict without a rerun. The code, logs, and preregistration text are unavailable during review, which prevents checking these consequential analysis choices. I would prefer a narrower paper centered on the exact baseline and robust observed-instance results unless these issues are resolved.
Questions for authors
-
Does the Q4 heterogeneity result persist when the null permits model-specific slopes and flexible difficulty-response curves? How many distinct exact d values occur, how many have multiple non-mirrored instances, and can those matched groups support a direct within-d test?
-
Please make the agent-specific fits central to the interpretation. Under a model allowing separate slopes and instance-effect distributions, which claims about random-like steepness and excess instance variation remain supported? The common fit's preregistration does not establish the validity of its assumptions.
-
Can you replace or supplement the d=0.1 claims with inference on the observed ranges, and provide an anonymous snapshot of the instance suite, outcomes, analysis, and frozen plan? For the intervention arm, would you either resolve the gate defect with a validated rerun or restrict the conclusion to the direct observation that agents select well when exact values are printed? These changes could materially improve my assessment.