- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
LUDKA Bench evaluates ML agents on 20 competition tasks designed to require task-specific insight. Each agent receives no guidance, a strategic hint, or a detailed description of a successful approach. Guidance improves aggregate performance, but agents still fall short of strong historical solutions on many tasks. A study with 23 experienced participants provides a human reference under the detailed-guidance condition, and execution examples illustrate difficulties in implementing and selecting solutions.
Critical strengths
-
S1. Varying the amount of supplied solution information is a useful benchmark design. It asks an important practical question: how much does knowing a promising approach actually help an agent solve the task?
-
S2. The tasks involve diverse forms of ML problem solving, and the evaluation preserves meaningful constraints on resources, feedback, and final submission selection.
-
S3. The human study and concrete execution traces add substantial value beyond an agent leaderboard. In particular, the human results support the usefulness of the supplied descriptions under the evaluation conditions.
Critical weaknesses
-
W1. Guidance effects are evaluated without an estimate of run-to-run uncertainty. Section 4.3 reports one run per task, information condition, and agent configuration. Keeping resources and feedback fixed makes the comparisons informative, but a single realization cannot show how stable the gains are when agent decisions, model training, and experimentation vary. This matters for interpreting both assistance effects and differences between systems. Repeated runs on a representative subset, with uncertainty on the within-task guidance gains, would substantially strengthen the main empirical conclusions.
-
W2. The robustness of task selection to the screening configuration is not established. The benchmark starts from 55 tasks, restricts eligibility to the 37 with public solutions, and removes tasks on which a Codex agent reaches the historical top three through a manually verified straightforward approach. Deliberately selecting difficult, insight-dependent tasks is reasonable for this benchmark. The unresolved issue is whether changing the screening model or reasonable implementation of the simple strategy would materially change the selected 20 tasks. Sharing the Codex harness with evaluated systems makes the relationship worth clarifying, but does not establish identical models or biased rankings. A clearer screening specification and a small sensitivity check would help distinguish broadly challenging tasks from configuration-specific failures; public-solution availability should be stated as a coverage constraint.
Questions for authors
-
Q1. Could the authors provide repeated runs for at least a subset of tasks and report uncertainty over the guidance effects? It would be useful to know whether the HINT-minus-NONE and SOLUTION-minus-NONE gains remain consistent across reruns under the same resource and feedback budgets.
-
Q2. How exactly were the HINT and SOLUTION descriptions constructed and standardized across tasks? Who wrote and checked them, what determined the boundary between strategic hints and implementation details, and was there a common checklist for fidelity and completeness? The shared reference solution and the example in Appendix C are helpful, but do not fully describe this process.
-
Q3. Which model and configuration powered the Codex screening agent, and how sensitive is task inclusion to that choice? Could the authors characterize the excluded tasks and, on a small subset, check whether a different screening agent would make similar inclusion decisions? Please also clarify how the public-solution requirement limits the intended scope of the benchmark.
-
Q4. Could the authors provide task-level guidance effects rather than only aggregate means? In particular, on which tasks does HINT capture most of the benefit of SOLUTION, and on which tasks is the additional detail in SOLUTION necessary? A per-task NONE/HINT/SOLUTION breakdown would make the benchmark substantially more diagnostic and help determine whether the aggregate gains are broad or driven by a small subset of tasks.
-
Q5. How do the authors assess the possibility of prior exposure to the benchmark tasks or their solutions? Since the benchmark is constructed from publicly released competitions and the final candidate pool consists of tasks with publicly available solutions, some models may have encountered the task statements, related discussions, or solution patterns during pretraining. Could the authors report the task release dates relative to the evaluated models, discuss any contamination checks that were performed, and clarify whether unguided trajectories were examined for distinctive overlap with public solutions? This would help establish how confidently the NONE condition can be interpreted as de novo strategy discovery.
2 comments
Q3. As described in our response to W2, the original screening used GPT-6 Astra with medium reasoning in Codex. The additional check used Claude Opus 5.5 with medium reasoning in Claude Code and identified no further tasks for exclusion from the final benchmark.
The screening excluded eight tasks where a verified straightforward approach reached the historical top three. For example, in Pixel Quest, the intended task involved navigation from visual observations, but the supplied seeds allowed the agent to reconstruct the maps and find paths using breadth-first search. Claude did not find a qualifying straightforward approach for Pixel Quest or Watermark Removal, both of which remained excluded based on the verified solutions found during the original screening. Appendix F.2 describes the procedure and example, while Appendix F.1 lists excluded tasks and their reasons for exclusion.
The public-solution requirement restricts the benchmark’s coverage to tasks with documented approaches that could be used to construct HINT and SOLUTION. Of the 55 catalogued tasks, 37 had public solutions and proceeded to screening. We have clarified this constraint in Section 3.1 and Appendix F.1.
Q4. We have added a per-task breakdown for all four agent configurations in Appendix B.3, Tables 6–9. For each task and information condition, we report the private score, historical rank, and normalized points.
The results show that the value of additional detail varies across tasks. Across the 80 task–configuration comparisons, HINT outperformed NONE in 49 cases and SOLUTION outperformed NONE in 52. SOLUTION outperformed HINT in 50. We have added this analysis to Section 5.1 and Appendix B.3.
Q5. We have added an analysis of competition and public solution dates relative to the models’ reported knowledge cutoffs in Appendix F.5. Some task and solution records predate these cutoffs, so prior exposure remains possible.
As an additional check, we reran NONE with rewritten task statements, metadata, and baselines, replacing visible task and competition identifiers while keeping the underlying data, objective, and evaluation procedure fixed. Across the four evaluated configurations, mean normalized scores decreased by 2.31–6.34 points. Changes in wording and run-to-run variation may also affect these results. We report the procedure and results in Appendix F.6.
We also examined generated code and available compressed reasoning traces from GPT-6 Astra/Codex NONE runs for reference code reuse and signs of solution recall. Automated screening was followed by manual review of flagged cases, with no confirmed instances of either. Together, these checks increase our confidence in the interpretation of the unguided results.
Thank you for your detailed review and for highlighting the need to assess run-to-run uncertainty and screening sensitivity.
W1. We agree that a single run per condition leaves the stability of the guidance gains unclear. We focused the repeat study on GPT-6 Astra (medium) with Codex because this configuration underpins most of our analysis of how agents use guidance and the conclusions drawn from it. We ran it two additional times under each information condition on all 20 benchmark tasks, keeping the resource budgets and feedback protocol fixed. Appendix B.4 reports the results for each task and run.
For each repetition, we computed the HINT–NONE and SOLUTION–NONE differences in normalized points within each task and averaged them across the 20 tasks. Across the three repetitions, HINT improved on NONE by 5.14 points (95% Student’s t CI: [3.72, 6.55]) and SOLUTION improved on NONE by 7.61 points (95% CI: [5.34, 9.88]). Both gains were positive in every repetition, and SOLUTION had the highest mean score in all three. These intervals quantify run-to-run uncertainty for Astra/Codex on the fixed benchmark. Estimating uncertainty in differences between agent configurations would require repeated runs of those configurations as well.
W2. We have clarified the screening procedure in Section 3.1 and Appendix F.2–F.3. Three international AI olympiad medalists first reviewed the available solutions for evidence of task-specific insights. We then screened the remaining tasks using GPT-6 Astra with medium reasoning in Codex, with one run per task. A task was excluded at this stage only if the agent reached the historical top three using a straightforward approach verified by the reviewers.
To assess sensitivity to the screening configuration, we repeated the agent screening using Claude Opus 5.5 with medium reasoning in Claude Code, using the same prompt and one run per task. Despite changing both the model family and the agent framework, this check identified no additional tasks for exclusion from the final benchmark. Claude did not find a qualifying straightforward approach for two tasks excluded by the original screening, Pixel Quest and Watermark Removal. Both remain excluded because the original screening had already produced verified solutions meeting the exclusion criterion. Overall, the check provides evidence that the final task set is stable with respect to this change in screening configuration.
We also clarified the coverage constraint in Section 3.1 and Appendix F.1: only the 37 of 55 catalogued tasks with publicly available solutions proceeded to screening, since these solutions were needed to construct the guidance conditions.
Q1. As reported in our response to W1 and in Appendix B.4, we ran Astra/Codex three times on all 20 tasks under the same resource and feedback budgets. Both HINT–NONE and SOLUTION–NONE gains were positive in all three runs. The 95% confidence intervals for the mean gains were [3.72, 6.55] and [5.34, 9.88] normalized points, respectively.
Q2. For each task, one of three international AI olympiad medalists manually wrote both HINT and SOLUTION from the same reference solution, and the other two reviewed both texts against it. We prioritized solutions published by the task authors, followed by public contestant implementations. When only a written solution was available, we used an LLM to convert it into code and accepted it as a reference only after checking the implementation and confirming a top-three result.
The same distinction was applied across tasks: HINT conveys the key idea while leaving the agent to develop the pipeline; SOLUTION describes the main steps of the approach while leaving implementation, debugging, and tuning to the agent. The reviewers checked that HINT accurately conveyed the intended idea and that SOLUTION preserved the key steps and provided enough detail to implement the approach. Depending on the task, the latter included data processing, training, validation, inference, postprocessing, and important implementation pitfalls.
We have clarified this process in Section 4.2 and Appendix C.1.