acceptodds
Under review as a conference paper at ICLR 2027

LUDKA Bench: Can AI Agents Solve Nonstandard Machine Learning Problems?

Abstract

Solving challenging, nonstandard problems in artificial intelligence and machine learning requires identifying useful structure, choosing an effective approach, and translating it into a working solution under limited time and experimentation. We introduce a benchmark comprising 20 of the most challenging tasks from international AI olympiads and national qualifying competitions from multiple countries to evaluate these capabilities in language models and agentic systems. Beyond measuring independent problem solving, the benchmark asks whether providing the right direction is sufficient for a system to solve a difficult task. We evaluate three levels of assistance: no guidance, a strategic hint, and a detailed description of an approach used in a solution ranked among the top three on the original private leaderboard. In our current evaluation, the strongest tested system achieves 18.5 mean rank without guidance and 8.0 with a detailed description of a reference approach. Competition-experienced human participants achieve a mean rank of 6.09 under the same evaluation conditions with detailed solution descriptions, on tasks they had not previously solved. Additional guidance improves aggregate performance, but even full descriptions leave a gap relative to these participants. These findings show that even detailed guidance based on a successful solution is often insufficient for agents to achieve a comparable result without further human assistance.

Primary area:
datasets and benchmarks
Keywords:
machine learning agents, strategy discovery, learning from hints, agent evaluation, benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BHHkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

LUDKA Bench evaluates ML agents on 20 competition tasks designed to require task-specific insight. Each agent receives no guidance, a strategic hint, or a detailed description of a successful approach. Guidance improves aggregate performance, but agents still fall short of strong historical solutions on many tasks. A study with 23 experienced participants provides a human reference under the detailed-guidance condition, and execution examples illustrate difficulties in implementing and selecting solutions.

Critical strengths
  • S1. Varying the amount of supplied solution information is a useful benchmark design. It asks an important practical question: how much does knowing a promising approach actually help an agent solve the task?

  • S2. The tasks involve diverse forms of ML problem solving, and the evaluation preserves meaningful constraints on resources, feedback, and final submission selection.

  • S3. The human study and concrete execution traces add substantial value beyond an agent leaderboard. In particular, the human results support the usefulness of the supplied descriptions under the evaluation conditions.

Critical weaknesses
  • W1. Guidance effects are evaluated without an estimate of run-to-run uncertainty. Section 4.3 reports one run per task, information condition, and agent configuration. Keeping resources and feedback fixed makes the comparisons informative, but a single realization cannot show how stable the gains are when agent decisions, model training, and experimentation vary. This matters for interpreting both assistance effects and differences between systems. Repeated runs on a representative subset, with uncertainty on the within-task guidance gains, would substantially strengthen the main empirical conclusions.

  • W2. The robustness of task selection to the screening configuration is not established. The benchmark starts from 55 tasks, restricts eligibility to the 37 with public solutions, and removes tasks on which a Codex agent reaches the historical top three through a manually verified straightforward approach. Deliberately selecting difficult, insight-dependent tasks is reasonable for this benchmark. The unresolved issue is whether changing the screening model or reasonable implementation of the simple strategy would materially change the selected 20 tasks. Sharing the Codex harness with evaluated systems makes the relationship worth clarifying, but does not establish identical models or biased rankings. A clearer screening specification and a small sensitivity check would help distinguish broadly challenging tasks from configuration-specific failures; public-solution availability should be stated as a coverage constraint.

Questions for authors
  • Q1. Could the authors provide repeated runs for at least a subset of tasks and report uncertainty over the guidance effects? It would be useful to know whether the HINT-minus-NONE and SOLUTION-minus-NONE gains remain consistent across reruns under the same resource and feedback budgets.

  • Q2. How exactly were the HINT and SOLUTION descriptions constructed and standardized across tasks? Who wrote and checked them, what determined the boundary between strategic hints and implementation details, and was there a common checklist for fidelity and completeness? The shared reference solution and the example in Appendix C are helpful, but do not fully describe this process.

  • Q3. Which model and configuration powered the Codex screening agent, and how sensitive is task inclusion to that choice? Could the authors characterize the excluded tasks and, on a small subset, check whether a different screening agent would make similar inclusion decisions? Please also clarify how the public-solution requirement limits the intended scope of the benchmark.

  • Q4. Could the authors provide task-level guidance effects rather than only aggregate means? In particular, on which tasks does HINT capture most of the benefit of SOLUTION, and on which tasks is the additional detail in SOLUTION necessary? A per-task NONE/HINT/SOLUTION breakdown would make the benchmark substantially more diagnostic and help determine whether the aggregate gains are broad or driven by a small subset of tasks.

  • Q5. How do the authors assess the possibility of prior exposure to the benchmark tasks or their solutions? Since the benchmark is constructed from publicly released competitions and the final candidate pool consists of tasks with publicly available solutions, some models may have encountered the task statements, related discussions, or solution patterns during pretraining. Could the authors report the task release dates relative to the evaluated models, discuss any contamination checks that were performed, and clarify whether unguided trajectories were examined for distinctive overlap with public solutions? This would help establish how confidently the NONE condition can be interpreted as de novo strategy discovery.

2 comments

Thank you for your detailed review and for highlighting the need to assess run-to-run uncertainty and screening sensitivity.

W1. We agree that a single run per condition leaves the stability of the guidance gains unclear. We focused the repeat study on GPT-6 Astra (medium) with Codex because this configuration underpins most of our analysis of how agents use guidance and the conclusions drawn from it. We ran it two additional times under each information condition on all 20 benchmark tasks, keeping the resource budgets and feedback protocol fixed. Appendix B.4 reports the results for each task and run.

For each repetition, we computed the HINT–NONE and SOLUTION–NONE differences in normalized points within each task and averaged them across the 20 tasks. Across the three repetitions, HINT improved on NONE by 5.14 points (95% Student’s tt CI: [3.72, 6.55]) and SOLUTION improved on NONE by 7.61 points (95% CI: [5.34, 9.88]). Both gains were positive in every repetition, and SOLUTION had the highest mean score in all three. These intervals quantify run-to-run uncertainty for Astra/Codex on the fixed benchmark. Estimating uncertainty in differences between agent configurations would require repeated runs of those configurations as well.

W2. We have clarified the screening procedure in Section 3.1 and Appendix F.2–F.3. Three international AI olympiad medalists first reviewed the available solutions for evidence of task-specific insights. We then screened the remaining tasks using GPT-6 Astra with medium reasoning in Codex, with one run per task. A task was excluded at this stage only if the agent reached the historical top three using a straightforward approach verified by the reviewers.

To assess sensitivity to the screening configuration, we repeated the agent screening using Claude Opus 5.5 with medium reasoning in Claude Code, using the same prompt and one run per task. Despite changing both the model family and the agent framework, this check identified no additional tasks for exclusion from the final benchmark. Claude did not find a qualifying straightforward approach for two tasks excluded by the original screening, Pixel Quest and Watermark Removal. Both remain excluded because the original screening had already produced verified solutions meeting the exclusion criterion. Overall, the check provides evidence that the final task set is stable with respect to this change in screening configuration.

We also clarified the coverage constraint in Section 3.1 and Appendix F.1: only the 37 of 55 catalogued tasks with publicly available solutions proceeded to screening, since these solutions were needed to construct the guidance conditions.

Q1. As reported in our response to W1 and in Appendix B.4, we ran Astra/Codex three times on all 20 tasks under the same resource and feedback budgets. Both HINT–NONE and SOLUTION–NONE gains were positive in all three runs. The 95% confidence intervals for the mean gains were [3.72, 6.55] and [5.34, 9.88] normalized points, respectively.

Q2. For each task, one of three international AI olympiad medalists manually wrote both HINT and SOLUTION from the same reference solution, and the other two reviewed both texts against it. We prioritized solutions published by the task authors, followed by public contestant implementations. When only a written solution was available, we used an LLM to convert it into code and accepted it as a reference only after checking the implementation and confirming a top-three result.

The same distinction was applied across tasks: HINT conveys the key idea while leaving the agent to develop the pipeline; SOLUTION describes the main steps of the approach while leaving implementation, debugging, and tuning to the agent. The reviewers checked that HINT accurately conveyed the intended idea and that SOLUTION preserved the key steps and provided enough detail to implement the approach. Depending on the task, the latter included data processing, training, validation, inference, postprocessing, and important implementation pitfalls.

We have clarified this process in Section 4.2 and Appendix C.1.

Q3. As described in our response to W2, the original screening used GPT-6 Astra with medium reasoning in Codex. The additional check used Claude Opus 5.5 with medium reasoning in Claude Code and identified no further tasks for exclusion from the final benchmark.

The screening excluded eight tasks where a verified straightforward approach reached the historical top three. For example, in Pixel Quest, the intended task involved navigation from visual observations, but the supplied seeds allowed the agent to reconstruct the maps and find paths using breadth-first search. Claude did not find a qualifying straightforward approach for Pixel Quest or Watermark Removal, both of which remained excluded based on the verified solutions found during the original screening. Appendix F.2 describes the procedure and example, while Appendix F.1 lists excluded tasks and their reasons for exclusion.

The public-solution requirement restricts the benchmark’s coverage to tasks with documented approaches that could be used to construct HINT and SOLUTION. Of the 55 catalogued tasks, 37 had public solutions and proceeded to screening. We have clarified this constraint in Section 3.1 and Appendix F.1.

Q4. We have added a per-task breakdown for all four agent configurations in Appendix B.3, Tables 6–9. For each task and information condition, we report the private score, historical rank, and normalized points.

The results show that the value of additional detail varies across tasks. Across the 80 task–configuration comparisons, HINT outperformed NONE in 49 cases and SOLUTION outperformed NONE in 52. SOLUTION outperformed HINT in 50. We have added this analysis to Section 5.1 and Appendix B.3.

Q5. We have added an analysis of competition and public solution dates relative to the models’ reported knowledge cutoffs in Appendix F.5. Some task and solution records predate these cutoffs, so prior exposure remains possible.

As an additional check, we reran NONE with rewritten task statements, metadata, and baselines, replacing visible task and competition identifiers while keeping the underlying data, objective, and evaluation procedure fixed. Across the four evaluated configurations, mean normalized scores decreased by 2.31–6.34 points. Changes in wording and run-to-run variation may also affect these results. We report the procedure and results in Appendix F.6.

We also examined generated code and available compressed reasoning traces from GPT-6 Astra/Codex NONE runs for reference code reuse and signs of solution recall. Automated screening was followed by manual review of flagged cases, with no confirmed instances of either. Together, these checks increase our confidence in the interpretation of the unguided results.

Reviewer_CFrBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces LUDKA Bench, a benchmark of AI Olympiad problems for LLM coding agents. They select a set of problems that require novel solutions and evaluate agents with varying amounts of guidance derived from a high-performing reference solution. They additionally evaluate humans when the highest level of guidance is provided. They find that further guidance improves performance and that agent failure modes qualitatively differ according to the guidance provided. They find that coding agent performance with high guidance is approaching human capabilities with the most recent LLMs but that none have exceeded human performance yet.

Critical strengths

The evaluation tasks are very strong in that olympiad problems are heavily vetted, sufficiently challenging, and relevant to LLM capabilities of interest. The authors’ filtering to the final set of tasks is also convincing in terms of task difficulty and relevance. The models’ diverse benchmark performance indicates the difficulty of the dataset.

The comparison to human and competitor evaluations shows that model capabilities at the guided tasks are just beginning to approach human performance, which makes these tasks of particular interest.

Critical weaknesses

The scope of the human evaluations limits what I can interpret from them. They are not run for the Hint or zero-guidance portions of the experiments, and these are the settings where I’m more interested in the comparison to human capabilities. The authors identify the need for a tighter selection of humans for these experiments, but their rationale doesn’t seem unique to these cases. They largely limit their interpretation of human results to showing that the solution guidance is useful, but this is already clear from the agent evaluation performance.

I’m somewhat unsure what the value is of the guided information conditions. It is not surprising or interesting that performance rises with stronger guidance when that guidance is very high-performing. I’m also unsure what exactly agent performance levels indicate about real-world capabilities. The Solution information condition doesn’t quite test agents’ ability to implement solutions or to follow user research instructions since the prompt is not an explicit specification that agents must follow and is too contrived to represent human instructions. The Hint condition fails to reflect any realistic setting.

Questions for authors

Can you provide a more direct definition of what is meant by the AI olympiad here and list the olympiads from which problems have been pulled? I would also appreciate some commentary on the novelty of this dataset in the wider ML research evaluation literature. Why couldn’t you have performed evaluations under different levels of guidance on existing datasets and gotten just as useful results?

Can you perform some checks to see if models are already aware of the research tasks and potential solutions? Analyzing chain-of-thought summaries, if they are saved in your logs, or prompting the agents about their awareness of the tasks up front would be helpful to check for data contamination.

Can you elaborate more quantitatively on how often agents diverge from the reference solution, using an LLM judge or some other method? This would provide clarity on the extent to which the agents interpret guidance as a strict specification.

Can you also elaborate on how often agents fail to select the best solution? Do agents have access to their historical submission code and performance in order to determine which solution is best? It could be the case that they are not selecting their best solution simply because it is unobservable rather than because of any failure to reason over solution candidates, especially if agents exceed their context window and compact it during the task. It is also not clear whether agents fail on this or on implementation itself, a distinction you raise in the introduction.

What real-world capability do you think each information condition measures? I’d appreciate a counterargument to what I write in the weaknesses.

Why do you restrict the number of agent submissions, and why is this number the same across all tasks? This limits the agents in comparison to the competitors, and I would imagine more submissions are more useful for some tasks than others.

Were reference approaches validated in your environment? The agent environment may differ from the original competition environment due to different GPUs or package versions, and this would be a strong check of compatibility.

6 comments

Thank you for your thoughtful review and for recognizing the value of the olympiad tasks and human comparison. Your questions prompted us to clarify what each guidance condition measures, examine how agents use the supplied approaches, and validate the reference implementations in our environment.

W1. The historical leaderboards already provide a comparison with human competitors who solved the original tasks without our guidance. We use them to evaluate agent performance under all three conditions, including NONE and HINT. The additional human study addresses a complementary question: how do humans and agents perform when given the same detailed solution description, resources, and feedback? We selected participants from the middle of the national qualifying pool, yet with SOLUTION they achieved a mean historical rank of 6.09, suggesting that the descriptions supported competitive solutions.

We agree that evaluating humans under NONE and HINT would allow a more controlled comparison across guidance levels. However, recruiting sufficiently large groups matched in ability and experience for each condition was not feasible, while having the same participants repeat tasks across conditions would introduce knowledge carryover.

W2. Solving a nonstandard ML problem involves identifying a promising idea, developing it into an effective approach, and implementing and validating that approach. Under NONE, the agent must perform all of these steps independently. HINT provides a key idea, leaving the agent to develop it into a complete solution. SOLUTION provides a more detailed description of a strong approach, allowing us to examine how effectively the agent can implement and validate it without human intervention.

While stronger guidance may be expected to improve performance, the magnitude of the gains shows how much agents benefit from a key idea and from additional implementation details, as well as what difficulties remain when a strong approach is supplied. The guidance texts are designed to convey solution information at different levels of detail rather than to reproduce a particular style of user instruction. These conditions help us study the stages of problem solving, although they do not perfectly isolate individual capabilities.

Q1. By AI olympiads, we refer to competitions in which participants develop ML solutions under round-specific time and resource constraints and submit predictions or executable code for evaluation. Final rankings are determined using private test data. We have clarified the competition format and sources in Section 3.1. The benchmark contains two tasks from the Russian qualifier for IAIO, five from the main rounds of NEOAI, and thirteen from AICC preparation contests. Appendix F.1 lists the source competitions and candidate tasks.

The guidance protocol can also be applied to existing benchmarks such as MLE-bench. However, LUDKA provides a complementary set of tasks selected for their reliance on task-specific insights: strong performance typically requires identifying and exploiting the particular structure of the problem, rather than combining many models and features into a general-purpose solution. This makes the benchmark particularly suitable for studying whether agents can develop a key idea into an effective approach and turn a detailed approach into a competitive solution. We have clarified this relationship to existing ML and research-agent benchmarks in Section 2.

Q2. We added contamination checks in Appendices F.5–F.6. First, we compared competition and public solution dates with the models’ reported knowledge cutoffs. Some solution records predate these cutoffs, so prior exposure remains possible.

Second, we reran NONE with rewritten task statements, metadata, and baselines, replacing visible task and competition identifiers while keeping the underlying data, objective, and evaluation procedure fixed. Across the four evaluated configurations, mean normalized scores decreased by 2.31–6.34 points. Changes in wording and run-to-run variation may also affect these results.

Finally, we examined generated code and available compressed reasoning traces for GPT-6 Astra with Codex. Automated screening was followed by manual review of flagged cases, with no confirmed instances of reference code reuse or solution recall.

Q3. We added an analysis of all 20 GPT-6 Astra/Codex runs under SOLUTION in Appendix D.5. An LLM agent inspected the execution logs and solution files, and a human checked the annotations against those materials. We compared each final submission with the key components explicitly described in the supplied guidance.

Across the 20 SOLUTION runs, 6 final submissions retained all key components, 10 used a partial or modified version of the reference approach, and 3 used a different approach. One run could not be classified. The partial or modified category includes submissions that changed or omitted components, or incorporated the reference approach into a broader solution.

Q4. We added a retrospective analysis of the 60 GPT-5.5/Codex runs in Section 5.3 and Appendix D.4. An earlier candidate achieved a higher private score than the final submission in 33 runs; selecting the best earlier candidate by private score would have improved the historical rank in 12 runs and resulted in one additional top-three finish.

During a run, agents receive public scores and can save candidate files and score records in the workspace. Saved files remain accessible after context compaction, although agents are responsible for maintaining their own history. Like agents, human participants select their final submissions using the available feedback and may not choose the candidate with the highest private score.

Private scores were unavailable during execution, so the retrospective results show how often a better earlier candidate existed, while our trajectory analysis also documents implementation and validation difficulties.

Q5. NONE evaluates an agent’s ability to solve an ML problem end-to-end, from identifying a promising direction to producing a working solution. HINT evaluates whether an agent can develop a partial idea into an effective approach, as when a researcher identifies a useful property of the data but leaves the method to be developed. SOLUTION evaluates whether an agent can turn a more detailed description of an approach into a competitive implementation through coding, experimentation, and validation.

We agree that the guidance texts are not representative of every instruction a user might give. They are designed to provide different amounts of solution information, allowing us to study these capabilities under controlled conditions. The conditions do not isolate the capabilities perfectly: each still requires the agent to make practical decisions throughout the task.

Q6. We use a common budget of 20 evaluations per task, based on the standard submission limit in the source competitions. This cap gives agents the same number of opportunities to obtain public feedback across tasks and guidance conditions.

We agree that additional submissions may be more useful for some tasks than others. The common cap provides a consistent evaluation protocol, though it does not account for these differences. We have clarified the budget and its motivation in Section 4.3 and Appendix A.

Q7. We use a common budget of 20 evaluations per task: up to 19 interactive submissions with public feedback and one reserved final evaluation. This matches the submission limit in the NEOAI and Russian IAIO selection rounds and provides the same feedback budget across tasks and guidance conditions. After exhausting the interactive budget, agents can continue training and validating locally within the time limit.

For the AICC tasks, submission limits varied across competitions. However, our inspection of the final leaderboards shows that on each of the 13 tasks, at least one top-three participant made fewer than 20 submissions. Moreover, on 11 of the 13 tasks, no participant on the leaderboard exceeded 20 submissions. This suggests that the common budget was sufficient to reach a competitive result on these tasks, while providing a consistent limit across agent conditions.

Q8. We reran a public reference implementation for each of the 20 tasks in the benchmark environment. All 20 produced evaluable submissions: 18 ranked in the historical top three, and the remaining two ranked fourth. These ranks were computed against the original competition leaderboards, including the submissions of the participants whose solutions we reproduced. We have added the task-level adaptations, scores, and ranks in Section 4.1 and Appendix E.2.

Thank you for your thoughtful commentary and apologies for a late response.

I appreciate your response for W1 and find the comparison to original competitors to be good though not directly comparable to your human evaluations. It would be helpful to make the point you make here on their comparability a little more explicit in the final version though.

I see your point for W2 that the magnitude of gains from different levels of guidance is interesting and find it compelling because it tells me which part of ML research is a capability bottleneck for agents. I do still struggle to see how I can relate these levels of guidance to anything specific in the real-world. When agents are given guidance in real-life, they are often told to strictly follow it, and strong capability is represented by how well they adhere to the guidance and perform conditional upon using that guidance. I think again you should be clear that these information conditions are purely just different amounts of information and not representative of agents' abilities to implement or follow a loose specification specifically, especially based upon your answer to Q3 showing agents diverge from the stated solution often.

For your response to Q1, it doesn't seem like these tasks are any more task-specific than other competition benchmarks like MLRC-Bench, so I don't think your tasks are necessarily more suitable for these experiments than prior work. Like I said in the strengths, I think these are strong tasks in general, so this is a minor concern.

I find the drop in performance following the contamination checks concerning. If it is true that rewording and run-to-run variation is what is affecting the changes in scores, this should be just as true for model/competition combinations where there was no potential exposure vs. those that did have potential exposure. Can you check whether this is the case? If not, I expect that agent solutions are contaminated even if the code and summarized traces don't indicate this. Some variance analysis showing that this difference in performance is not unlikely would be helpful too, though I'm not sure if there's a convincing way to do this.

Thank you for your responses to all other questions. They largely satisfied my concerns. You have addressed many of my main concerns, but I am still worried about this contamination issue. If you can provide additional analyses of that like I suggested or further argumentation about why this isn't concerning, that would be helpful.

Reviewer_1jwkOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The work introduces Ludka-Bench which is a 20-problem dataset for evaluating AI agents to solve complex ML problems (of olympiad scale). The authors evaluate three levels of guidance provided to the agents, including no-hint, some strategic guidance, and detailed description of the solution which is top-ranked. The work evaluated 2 GPT models (5.5, Astra) and 1 claude model (sonnet 5) along with their native harnesses such as codex, claude code. Overall, the work concludes by observing that mid-level guidance (strategic) improves performance but the highest level of guidance does not close the gap with the top participants solution.

Critical strengths
  1. The idea of obtaining complex problems for AI agents which challenges them for hours, and also weigh on not just final solution but also approach due to time and compute constraints is becoming relevant in current times.
  2. The work involved 23 human participants to obtain the human-graded solutions which is kind of obtaining "gold-labels" in ML terms
Critical weaknesses

This is a benchmark paper and a usual understanding is either a nice presentation of the (i) problem and data, (ii) well-defined quantitative metrics to properly reproduce and rank applicants. A benchmark would be useful only, if it has sufficient description of data and metrics such that potential models will be using it to get their position. Here, there is 20 samples in the benchmark and problem is defined but no focus is put on explaining why this data is complex and why current models may not be able to solve it well. Second, the paper has little to none effort devoted to quantify metrics to measure behaviors. There is a qualitative Sec-5.3 which does not warrant any reproducibility at all. I have some additional issues with the paper which are listed below

  1. The distinction in related-work is aiming to paint a picture that proposed benchmark would differentiate poor approach in terms of failing to translate idea to action (line 101-102), however, the reviewer could not find in the paper any quantified version of it
  2. The arguments for data selection is loose. For example, line 154-156, giving instruction to codex that use straight-forward approach does not warrant any guarantee of it doing so. The reasoning of these models is hidden, and there is no evidence that it would just instruction-follow this verbatim. In other words, this steps needs a serious re-work.
  3. line 158-159, the process is mentioned very hand-wavy about verifying solution to infer whether task-specific instructions were used. This needs a concrete process outlined somewhere
  4. The paper has space dedicated to trivial equations such as (1)-(2), which instead could be utilized for relevant details such as metrics and better distinction of models evaluated.
Questions for authors

please see the weakness section

3 comments

Thank you for your careful and constructive review. Your feedback prompted a substantial revision of the manuscript, with changes throughout the paper and its appendices.

Benchmark specification and task difficulty. We have substantially expanded the description and supporting materials to make the benchmark easier to reproduce and interpret. Section 3.1 and Appendix F now detail the source competitions, candidate catalog, screening decisions, and full screening prompt. Section 4 and Appendices A and E.1 specify the agent configurations, resource and time limits, submission interface, feedback, and final selection procedure.

Appendix B.3 (Tables 6–9) reports the private score, historical rank, and normalized points for each task, agent configuration, and information condition. We also repeated the GPT-6 Astra/Codex evaluation twice on all 20 tasks under each information condition, keeping the resource budgets and feedback protocol fixed. Appendix B.4 reports the results of all three runs and estimates uncertainty in the gains from guidance. We validated a reference implementation for every task in the benchmark environment and report the adaptations, scores, and ranks in Appendix E.2.

Task difficulty. The 20 benchmark entries are complete ML competition tasks, each involving data analysis, method development, experimentation, and final submission under time and resource constraints. We selected tasks for which strong solutions typically depend on identifying and exploiting task-specific structure.

Section 3.1 and Table 1 illustrate several forms of this difficulty. Broken BERT requires reconstructing damaged token embeddings under restrictions on fine-tuning and external models. Classifier Classifier requires identifying what each neural network recognizes without access to its original training data. Lost Interpreter requires inferring unknown instruction semantics from example programs and their outputs before executing new programs. Appendix F details how we screened tasks for these demands.

Quantitative diagnostics. We measured guidance gains within each task and averaged them across the 20 tasks for each Astra/Codex run. Both gains were positive in all three runs. Across runs, HINT improved on NONE by 5.14 normalized points (95% CI [3.72, 6.55]), and SOLUTION improved on NONE by 7.61 points (95% CI [5.34, 9.88]). These intervals capture run-to-run uncertainty for Astra/Codex on the fixed benchmark (Appendix B.4).

We also analyzed the execution logs and solution files for all 60 Astra/Codex trajectories from the initial run. An LLM agent inspected the materials, and a human checked the annotations. The agent implemented or tested the reference idea on 8 of 20 tasks under NONE and 17 of 20 under each guided condition. Under SOLUTION, 6 final submissions retained all key components, 10 used a partial or modified approach, and 3 used a different approach. One case could not be classified. The partial or modified category includes changes, omissions, and cases where the agent incorporated the reference approach into a broader solution. Appendix D.5 reports the annotation procedure and task-level results.

Finally, we added a retrospective analysis of final submission selection in Section 5.3 and Appendix D.4. Across 60 GPT-5.5/Codex runs, an earlier candidate achieved a higher private score in 33 runs. Selecting the best earlier candidate by private score would have improved the historical rank in 12 runs and produced one additional top-three finish. Both agents and human competitors selected their final submissions without access to private scores. Appendix D.4 breaks down the results by information condition, and the cases in Section 5.3 illustrate the decisions and experiments behind individual runs.

W1. We have revised Section 2 to clarify how LUDKA studies the distinction between finding an approach and turning it into a competitive solution. Solving a nonstandard ML problem involves identifying a promising idea, developing it into an effective approach, and implementing and validating that approach. Under NONE, the agent must perform all of these steps independently. HINT provides a key idea, leaving the agent to develop it into a complete solution. SOLUTION provides a more detailed description of a strong approach, allowing us to examine how effectively the agent can implement and validate it without human intervention.

Guidance made agents more likely to try the reference idea, but its key components often did not carry through to the final submission. Both agents and human competitors selected which submission would be scored on the private test set without access to private scores. Across 60 GPT-6 Astra/Codex trajectories, the agent implemented or tested the reference idea on 8 of 20 tasks under NONE and 17 of 20 under each guided condition. Under SOLUTION, 7 of 20 final submissions retained all key components, including one that extended the approach. Nine modified or omitted at least one key component, three followed a different approach, and one could not be classified (Section 5.3 and Appendix D.5).

We also quantified how much better final selection could have helped among the candidates the agents had already produced. Across 60 GPT-5.5/Codex runs, selecting the best candidate in hindsight would have improved the private score in 33 runs and the historical rank in 12. It would have yielded just one additional top-three finish, raising the SOLUTION count from 5 to 6 of 20 tasks. Thus, even with access to private scores for final selection, the candidates produced under SOLUTION would still have fallen outside the historical top three on 14 tasks (Section 5.3 and Appendix D.4).

We have also expanded Section 5.1 and Appendix B.3 to report results under all three conditions for every task and agent configuration. For example, GPT-6 Astra reaches the historical top three on 6 of 20 tasks under NONE and 11 of 20 under SOLUTION. Thus, even with a description derived from a top-three solution, the agent remains outside the top three on 9 tasks.

In addition, Section 5.3 and Appendix D.4 now quantify final candidate selection: across 60 GPT-5.5/Codex runs, an earlier candidate achieved a higher private score than the final submission in 33 runs. The guidance comparisons and this retrospective analysis measure different aspects of the process, while the trajectory examples provide further detail on the difficulties encountered during implementation and validation.

W2. We agree that prompting an agent to use straightforward approaches does not guarantee that it will do so. We have clarified the screening procedure in Section 3.1 and Appendix F.2–F.3: the prompt was used to guide the search, while exclusion depended on the implemented solution and its performance.

When a screening run reached the historical top three, the reviewers inspected the implementation to determine whether it used a task-specific insight. If it did, they examined the highest-scoring alternative that avoided that insight. A task was excluded only when a manually verified straightforward approach reached the top-three threshold.

We also repeated the screening using Claude Opus 5.5 with medium reasoning in Claude Code, using the same prompt and one run per task. This check identified no additional tasks for exclusion from the final benchmark. Claude did not find qualifying approaches for Pixel Quest and Watermark Removal, which remained excluded because the original screening had already produced verified solutions meeting the criterion.

W3. We have detailed the verification procedure in Appendix F.2 and summarized the decision rules in Table 21. When a screening run reached the historical top three, the reviewers inspected its implementation to determine whether the approach exploited a task-specific property of the data, model, metric, or rules. If the top-scoring implementation used such an insight, they examined the highest-scoring alternative that avoided it. A task was excluded only if a manually verified straightforward implementation reached the top-three threshold.

Appendix F.2 also provides a concrete example. In Pixel Quest, the reviewers found that the supplied seeds allowed the maps to be reconstructed, after which breadth-first search produced a top-three solution. This shortcut was treated as a straightforward approach available from the task materials, so the task was excluded.

W4. We have streamlined Section 4.5 by moving the equations to Appendix B.1 and explaining normalized points, historical ranks, top-k placements, and guidance gains in the main text. Appendix B.4 also reports the results for each task and guidance condition.

We have made the evaluated configurations easier to distinguish: Table 3 specifies the model, reasoning setting, and agent framework for each result. Further execution details are provided in Appendix A.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.