acceptodds
Under review as a conference paper at ICLR 2027

Codoku: Renewable Program-Reasoning Challenges for Frontier Coding Agents

Abstract

Existing program-reasoning benchmarks ask large language models to predict a program's behavior on a given input. Coding agents break two assumptions on which these benchmarks rest: an agent can recover the answer by executing the program instead of reasoning about it, and fixed task sets drawn from existing programs are increasingly exposed to contamination, yet costly to renew. We introduce Codoku (code sudoku), a renewable benchmark in which a solver fills typed cells in a partial program to satisfy global static and dynamic constraints, such as a prescribed control-flow graph and execution path. Because a partial program cannot be executed and valid fillings are sparse in an exponentially large space of interdependent choices, neither tool use nor enumeration can substitute for program reasoning. Puzzles are synthesized from scratch via semantic reification, so fresh puzzles of controllable complexity can be generated on demand, each with a witness that guarantees solvability. We evaluate five frontier models on 300 puzzles through a coding agent free to use any tool within a fixed budget. Small puzzles already challenge open-weight models, whereas even proprietary models solve only about half of the large ones. Codoku thus offers a renewable testbed for program reasoning that can keep pace with rapidly improving coding agents.

Primary area:
datasets and benchmarks
Keywords:
program reasoning, coding agents, benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gwGTOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Codoku - a novel benchmark for assessing the performance of coding agent+harness combinations. The puzzle synthesis is performed using semantic reification algorithm for random program synthesis. Part of the program is then masked with symbols corresponding to their type and the agents are supposed to fill these masks in a way the checks out with several tests. The authors evaluate several agents to show that this benchmark is challenging.

Critical strengths
  1. The proposed benchmark allows synthesizing arbitrary many problems and is not saturated out of the box.
  2. The authors evaluate several state of the art open and closed source llms.
Critical weaknesses
  1. The codoku puzzles are inherently CSPs rather than programming problems, which are inherently much more open ended. The authors position these problems as benchmarks for coding agents, which may be misaligned.
  2. I think those would be very difficult to solve even for humans - from the few examples presented in the appendix. Would the conclusions from performance on these problems generalize?
  3. While the authors have provided analysis of the failure modes and techniques employed by various agents, the analysis could be broader, including more cherry picked failure analysis, and even more importantly - cumulative curves of the agent resource use (time / tokens / money ) per solutions attained - maybe all the problems will get solved if you give some of the agents enough time (overwhelming majority of failures are exhaustion).
  4. While the paper provides an artifact - the codoku benchmark, the overall novelty of the techniques used is difficult for me to assess, as I am not closely familiar with program synthesis literature.
Questions for authors
  1. Do differnt agents manage to solve an overlapping core of problems? Or could it be that different model/harness combination manage to solve different puzzles?
  2. The way the problem is formulated with the typed cells being enums / integers for the most part, could these problems be solved with a SAT solver or a SAT-aware neurosymbolic approach?
2 comments

Answer to W1

W1: The codoku puzzles are inherently CSPs rather than programming problems, which are inherently much more open ended. The authors position these problems as benchmarks for coding agents, which may be misaligned.

Codoku is indeed a CSP, but the domain is programs and it evaluates constraint-guided program reasoning.

Codoku concerns the static and dynamic semantics of a program: a valid solution must compile, match the control-flow graph, follow the execution path, and produce the expected return value on concrete inputs under a constant budget.

Formulating and solving these constraints combines:

  • Structure reasoning: choosing constants, identifiers, operators, functions, and control-flow targets for each cell.
  • Execution reasoning: determining whether those choices produce the required control flow, program states, and output.

Codoku is therefore a benchmark for a specific capability of coding agents, i.e., constraint-guided program reasoning: an agent must reason locally about each cell and globally about their inter-dependencies, infer candidate structures, and revise it when its behavior violates our constraints. We will clarify this scope in the paper.

Answer to W2/Q2

W2: I think those would be very difficult to solve even for humans - from the few examples presented in the appendix. Would the conclusions from performance on these problems generalize?

Q2: The way the problem is formulated with the typed cells being enums / integers for the most part, could these problems be solved with a SAT solver or a SAT-aware neurosymbolic approach?

Codoku is challenging but tractable through constraint-guided program reasoning: the best-performing agent solves 50% of large puzzles, while three direct-search and -solve baselines solve none under the evaluated budget.

Finite cell domains permit SAT/SMT encodings. However, the challenge is not just assigning enum or integer values: the encoding must also capture the program's structure, control flow, data flow, and execution constraints. Following reviewers’ suggestions, we built three additional baselines (our agent experiment in Section 3 already demonstrates an SMT-aware neurosymbolic approach):

  • Type-aware random search: sample values from each cell’s allowed domain and check the candidate.
  • Full-puzzle SMT encoding: jointly encode program structure and execution constraints in Z3.
  • Template-based SMT encoding: enumerate assignments to non-constant cells, then solve a constants-only puzzle through Z3 encoding.

Each baseline attempted all 300 puzzles with checker access, unlimited candidate checks, and the paper's one-hour time budget per puzzle.

BaselineSolvedTimeoutExhaustedUnsupportedError
Type-aware random search0300000
Full-puzzle SMT encoding02842140
Template-based SMT encoding0298002

Across 900 solver-puzzle runs, none succeeded, and 98% timed out, consistent with Codoku’s design to make undirected search (without program reasoning) ineffective.

But Codoku puzzles remain tractable through reasoning about local program states and decomposing puzzles into smaller subproblems. Successful agents combine this local reasoning with algebra, search, constraint solving, repair, and other strategies, as presented in Section 4.1. The best-performing agent solves 50% of large puzzles, indicating a level of difficulty comparable to TerminalBench and FrontierCode.

Our goal is to elicit, thus evaluate, the program reasoning capabilities of agents. We agree with the reviewer that whether performance transfers to broader coding tasks is an interesting question, which we leave for future work.

Answer to Q1

Q1: Do differnt agents manage to solve an overlapping core of problems? Or could it be that different model/harness combination manage to solve different puzzles?

Yes, agents share a small solvable core, particularly among small puzzles: 23 are solved by all five models, compared with 11 for medium puzzles and only 3 for large puzzles. We will add a Venn diagram to illustrate these overlaps in the revision.

For harnesses, we selected Pi because it is lightweight, without vendor-specific prompting, orchestration, or tools that could obscure the underlying model’s reasoning or steer it along potentially biased paths. Different model-harness combinations may solve different sets of puzzles; investigating this further would be an interesting future work. We will clarify this rationale in the revision.

Answer to W3

W3: While the authors have provided analysis of the failure modes and techniques employed by various agents, the analysis could be broader, including more cherry picked failure analysis, and even more importantly - cumulative curves of the agent resource use (time / tokens / money ) per solutions attained - maybe all the problems will get solved if you give some of the agents enough time (overwhelming majority of failures are exhaustion).

The failure categories and analysis in Table 1(b) and Appendix A.4 already cover all 1,500 runs, including their termination reasons and fine-grained checker results.

Figure 3 and Appendix A.3 report average resource usage; Lines 442–443 and Appendix A.5 present the budget-extension experiment. We doubled all three limits (two hours, $30, and 256 requests) for the best-performing agent, GPT 5.6 Sol, on the large puzzles. Gains were modest and turned to converge: only 8 of 56 previously unsolved puzzles were solved, while the remaining reached the two-hour limit. Inspection suggests that ineffective subproblem decomposition and strategy selection, account for most failures and limit the benefits of larger budgets.

In the revision, we will move some content into the main text and add a cumulative curve of agent resource uses. We will also release all data.

Reviewer_pQeYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

An automatically generated benchmark for evaluating program reasoning in coding agents by requiring them to fill in the masked portions of code given the constraints of input, output and execution path. The base code is generated through semantic reification (Chopra et al., 2026), which is acted upon by a puzzle-synthesiser (proposed by the authors) to mask some tokens. The paper evaluates five frontier models on 300 puzzles (partial codes) through a coding agent.

Critical strengths
  1. The paper identifies key weaknesses in existing program reasoning benchmarks, including contamination and the redundancy of execution-prediction tasks when coding agents have access to tools such a compiler/executor. An agent can simply execute the program rather than reason about its behavior. The paper addresses this by masking parts of a program, making it non-executable, and requiring models to recover the missing tokens from a large search space while satisfying the given constraints. (However, although this approach forces models to engage with the code without relying on execution, it does not, by itself, establish that the task poses a genuine program-reasoning challenge, as discussed in the Weaknesses section).

  2. No evaluated model saturates the benchmark; even small partial codes leave substantial headroom, and large partial codes top out around 54%. It also separates proprietary and open-weight models and shows that resource scaling helps only marginally.

  3. The paper uses a range of relevant metrics to characterize problem difficulty. These metrics appear to capture differences in reasoning demands, as reflected in the models' accuracy across problem instances.

Critical weaknesses
  1. Limited Algorithmic Novelty: The idea is not completely new. It combines known ingredients like masked-code filling (https://arxiv.org/pdf/2608.03691), constraint satisfaction (https://arxiv.org/html/2506.06052v1),SMT-backed generation (https://aclanthology.org/2023.findings-emnlp.248/), execution-path constraints (https://arxiv.org/html/2511.18288v1), into a useful benchmark format for coding agents. The novelty is mainly in the benchmark formulation and renewable generation pipeline, not in a new algorithmic method.

  2. Misalignment Between Task and Evaluation Objective: Constraint Solving ability using tool guided search is conflated with program reasoning. Many puzzles may reduce to algebraic/SMT/search-based constraint solving over synthetic code as shown by the results of the paper itself (Algebraic strategy being highly used by most models). It is not clear whether agents are solving the puzzle semantics, or learning the checker’s implementation details or solving an algebraic CSP. If the checker source is always available, agents may optimize against checker behavior rather than the abstract task. Enough automated search plus checker feedback may solve some puzzles with limited semantic understanding of the program. The synthetic nature of programs further aggravate this problem. Real code has names, abstractions, APIs, design patterns, comments, libraries. Codoku has symbolic variables, weird constants, arithmetic expressions, CFG annotations which brings the Codoku puzzles closer to CSF formulations rather than real world programs. So, strong performance on Codoku does not necessarily indicate stronger real-world program comprehension.

  3. Missing details on dataset generation and design choices: Paper mentions a three-stage pipeline to generate the dataset, consisting of witness synthesis’, ‘puzzle construction’ and ‘solution verification’ but significant details are missing under these sections.

In the ‘witness synthesis’, CFG and EP creation get a single sentence (“adding forward branches and loop back edges, then walk it”). The paper doesn’t describe the generation algorithm for CFG and EP. The paper does not state how many distinct CFGs were generated, whether CFGs are shared across puzzles, or how many puzzles derive from each CFG. It doesn’t say whether duplicates or near-duplicates were filtered out.

In the ‘puzzle construction’ section, the paper reports the number and type distribution of masked tokens (Table 2), but not how they arise. Policy to select a token for masking and the masking rate (fraction of maskable tokens masked) is not described. Cells that are weakly constrained, such as those in dead blocks, are acknowledged as a limitation but are not measured. A better version would use multiple input-output examples and execution paths so that all blocks are dynamically covered. Secondly, the puzzles are generated by a synthesizer with controlled templates which may result in generating similar or isomorphic instances. The paper does not report a semantic-diversity.

In the ‘solution verification’ section, the paper mentions that feedback is given in response to each specific failure but no information is provided about why this design choice is selected over alternatives like giving a global aggregate or binary feedback when the final solution is incorrect. The paper doesn’t mention any ablations on the feedback mechanism.

  1. Renewability is narrower than claimed: Contamination-resistance claim is not fully established at the semantic level. Renewability reduces the likelihood that exact puzzle instances were seen during training, but it does not guarantee that generated puzzles are semantically or structurally novel.
Questions for authors
  1. Partial programs with holes completed against a specification are the setting of program sketching (Solar-Lezama et al., 2006) and syntax-guided synthesis, and the generator is semantic reification (Chopra et al., 2026). Please state precisely what Codoku contributes beyond these. Please also add these lines of work to the related-work section.

  2. Please specify the CFG and execution-path generation procedure and its parameters for each profile, and the criterion used to select statements and tokens for masking. Also mention the semantic diversity of the puzzles generated.

  3. Provide ablations showing how much each constraint (constant table, execution path, CFG, multiple input-output examples) contributes to difficulty. Secondly, holding the witness program fixed, how does the rate change as the masking ratio varies? Which factor drives difficulty: cell count, path length, or loop iterations? How does limited tool availability intended to force maximal native program reasoning impacts the accuracy.

  4. Please report a non-LLM baseline under the same time budget. Candidates include a whole-puzzle SMT encoding, a symbolic-execution-based solver, and checker-guided backtracking or random search. If such a solver handles many puzzles, the benchmark measures solver orchestration more than program reasoning; if it fails, that substantially strengthens the central claim.

  5. In the case study, Phases 4–5 (modular arithmetic, factoring, and about 96 million random samples) account for about 71% of run time and 61% of cost. This suggests difficulty may stem largely from inverting the checksum computation rather than from control- or data-flow reasoning. Could you report solve rates on variants without checksum obfuscation, or otherwise quantify how much of the difficulty is attributable to the checksum arithmetic?

  6. To strengthen the contamination hypothesis, please construct a masked-code-regeneration baseline from existing real-world benchmarks, (preferably the programs for algebraic constraint satisfaction) using the same typed-cell masking, checker, budget and agents, and report performance.

2 comments

Answer to Q1/W1

Q1: … what Codoku contributes ...

W1: Limited Algorithmic Novelty: … not completely new. It combines known ingredients ….

Codoku contributes a renewable benchmark for evaluating the program reasoning capabilities of frontier coding agents. Its novelty lies in its formulation of program reasoning tasks as constrained program-synthesis problems and in its introduction of a pipeline that generates such renewable puzzles and verifies solutions. It does not aim to propose novel algorithms or solvers for program synthesis. Rather, it leverages an existing program synthesis technique, i.e., semantic reification, as its initial step to generate a witness solution for subsequent puzzle construction. The checker uses standard program analysis techniques to verify solutions; it is independent of witness solutions and accepts any program (beyond the witness solution) that satisfies the puzzle’s semantic constraints.

We will add a “Novelty and contributions” paragraph in the revision for clarification and add works the reviewer mentioned in “Related Work”.

Answer to W2

W2: Misalignment Between Task and Evaluation Objective: …

We clarify that algebra and SMT do support the program reasoning capability that Codoku is designed to evaluate.

The reviewer is right that many Codoku puzzles may reduce to algebraic/SMT/search-based constraint solving problems. However, this reduction requires formulating and decomposing the puzzle’s semantic constraints through jointly modeling the structure (cell choices) and the execution (control/data flow). This modeling process is unavoidable and integral to the reasoning capability we evaluate, because incorrect models yield incorrect solutions. Algebra, SMT, and other strategies are therefore legitimate techniques after the problem is correctly formulated. This is analogous to a mathematics exam that permits calculators: the student must formulate the problem correctly, while the calculator performs the calculation.

The checker uses standard techniques to parse, run, and trace the submitted solution. It is independent of valid solutions and does not supply any valid synthesis information. Knowing the checker does not tell an agent which identifier belongs in a cell, which operator satisfies a branch, or how to satisfy the constant budget.

Answer to Q4

Q4: Please report a non-LLM baseline under the same time budget. …

We evaluated three non-LLM baselines on all 300 puzzles, using the paper’s one-hour budget per puzzle:

  • Type-aware random search: sample values from each cell’s allowed domain and check the candidate.
  • Full-puzzle SMT encoding: jointly encode program structure and execution constraints in Z3.
  • Template-based SMT encoding: enumerate assignments to non-constant cells, then solve a constants-only puzzle through Z3 encoding.

All baselines had checker access and could make unlimited attempts within the time budget.

BaselineSolvedTimeoutExhaustedUnsupportedError
Type-aware random search0300000
Full-puzzle SMT encoding02842140
Template-based SMT encoding0298002

None of the 900 solver-puzzle runs succeeded; 98% timed out. Random search checked tens of thousands of type-valid candidates per puzzle. The full-puzzle SMT baseline encountered encoding and solving bottlenecks, while template-based SMT spent its budget enumerating templates.

This is consistent with Codoku’s design to remove straightforward search or solve approaches without program reasoning. However, our strategy analysis in Section 4.1 shows that agents can solve some puzzles by integrating reasoning about local program states with search, SMT solving, algebraic methods, and other strategies.

Answer to Q2/W4

Q2: ... semantic diversity ..

W4: Renewability …

We acknowledge that similar CFGs or program structures may appear in training data, a challenge shared by all benchmarks. However, Codoku’s exact programs are unseen, and semantically equivalent programs are highly unlikely to appear in training data.

Codoku uses CFGs, execution paths (EPs), and input-output examples as proxies for semantic diversity. For a fixed CFG, Codoku can sample distinct EPs with different input-output examples, yielding semantically distinct puzzles.

For the 300 evaluated puzzles, Table 1(a) summarizes their static and dynamic semantic characteristics, with further details in Table 2. We also measured structural and EP diversity across these puzzles as proxies for semantic diversity.

ProfileDistinct CFG edge setsMost frequent CFG countDistinct EPsMost frequent EP count
Small6163019
Medium964873
Large1001974

The table indicates that the small and medium puzzles reuse CFG shapes, whereas large puzzles do not. Even where CFG shapes are reused, the programs, expressions, constants, and checksums differ; all 300 puzzle files are distinct as well. We will include this analysis in the revision.

CFG generation, EP sampling, and statement masking are all randomized procedures. In the revision, we will describe these procedures and disclose the key hyperparameters. We will also release the source code upon publication.

Answer to Q5

Q5: … checksum computation …

Computing a checksum is a standard way to summarize a program's final state. In Codoku, it is a necessary correctness constraint: a weak checksum, such as a simple sum, can accept incorrect programs because different final states can produce the same checksum. To limit complexity, Codoku uses ordinary integer arithmetic and bitwise operations rather than a dedicated checksum algorithm such as CRC32, for example:

def _in_global_chksum(addend, c0, c1, c2, c3, c4, c5, c6, c7, c8):
    y = 10193773499649408735
    y = y ^ 7186 * c0
    y = y + 4251 * c1
    y = y - 2085 * c2
    y = y + 3177 * c3
    y = y | 9575 * c4
    y = y // (8688 * c5 | 1)
    y = y >> clamp(2901 * c6, 0, 64)
    y = y & 51 * c7
    y = y ^ 9611 * c8
    return addend + y

However, we agree that these operations may make checksum inversion costly, so following the reviewer's suggestion, we conducted a targeted ablation using GPT 5.6 Sol on 10 randomly selected, previously unsolved large puzzles. We replaced the original checksum with a sum-based checksum and kept the original evaluation budget (1h, $15, and 128 requests). The agent solved 2/10 puzzles, suggesting that checksum arithmetic was not the primary bottleneck in these cases. We will clarify our choice of checksum arithmetic in the revision.

Answer to Q3/Q6

Q3: ... ablations showing how much each constraint ...

Q6: ... masked-code-regeneration baseline from existing real-world benchmarks ...

We reiterate that Codoku’s central contribution is a renewable benchmark for evaluating program reasoning capabilities, not a mechanism for making puzzles maximally difficult. Our evaluation results support this goal: successful agents combine local program reasoning with other strategies (Section 4.1), while leaving room for further performance gains (Section 3). In Codoku, the CFG, execution path, constant table, and input-output examples represent complementary aspects of program semantics, rather than factors to tune difficulty.

Although larger profiles are empirically harder, we do not claim that difficulty increases monotonically with any individual factor. We agree that the requested ablations, including the matched real-world baseline, would clarify how these factors affect difficulty and the contamination hypothesis, and could inform improvements to Codoku's design. However, a full evaluation of even one proposed ablation would cost approximately $4,000, comparable to our current experiment and beyond our available computational budget. We will prioritize controlled, cost-effective ablations in our future design and development of Codoku 2.

Although we do not have quantitative evidence, our experience suggests that path length and loop iteration count, which control dynamic semantics, pose greater challenges than CFG size or cell count. The constant pool helps prevent agents from exploiting trivial values (e.g., 0 and 1) or special constants to neutralize certain statements and force the required return value.

Reviewer_HJ1iOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Codoku is a clever approach to benchmark LLM and agent harnesses that have access to tools on how well they can reason about code. Asking an agent or LLM what a program will produce on a set of inputs doesn't require reasoning if the agent can just invoke a code execution tool to determine exactly what happens. Codoku generates programs and masks out parts of the program, preventing simple execution to determine what a program does. Codoku requires the LLM/agent to fill in the missing pieces of a program in order for it to output a required return value, and with constraints requiring the code to execute a specific flow through the code blocks.

Critical strengths

The paper identifies the flaw of LLM benchmarks for reasoning about programs that just require a program to compute what the output is for a specific input. Agents/LLMs that have tool access can score perfect by just executing the code and not reasoning at all about the code.

The paper shows how the difficulty of the generated tasks can be measured

The approach allows for cheap generation of new test sets, so previous runs of the Codoku benchmark that get added to the train set don't contaminate subsequent runs on newly generated test sets.

Critical weaknesses

It's unclear if this benchmark is better at distinguishing program reasoning abilities, how well does it correlate/compare with other code benchmark methods like predicting the results of program execution without access to an execution environment.

The ability of the LLM to use a constraint solver seems like a hole, similar to running a program execution benchmark on an LLM/agent that has access to a code execution tool.

Questions for authors

How well does this metric correlate/compare to other program reasoning approaches?

Does this approach better predict program reasoning ability than a simple 3-SAT type benchmark? I admit it seems like it would be, but comparisons to other approaches are sparse.

2 comments

Answer to Q1/W1

Q1: How well does this metric correlate/compare to other program reasoning approaches?

W1: It's unclear if this benchmark is better at distinguishing program reasoning abilities, how well does it correlate/compare with other code benchmark methods like predicting the results of program execution without access to an execution environment.

Codoku does not replace these benchmarks, nor does it claim universal superiority over them. They measure different settings and complement each other.

Execution prediction starts with a complete program and asks for an output or an input. Codoku starts with a partial program and asks for a completion that satisfies the input-output examples, and other syntactic and semantic constraints (e.g., CFG, execution path, loop iterations). This combines:

  • Static, structure reasoning: choosing constants, identifiers, operators, functions, and control-flow targets for each cell.
  • Dynamic, execution reasoning: determining whether those choices produce the required control flow, program states, and output.

Execution prediction primarily focus on a fixed program’s forward/backward execution, i.e., the latter case, but Codoku concerns both jointly: an agent must reason locally about each cell and globally about their inter-dependencies, infer candidate structures, and revise it when its behavior violates our constraints.

Answer to Q2

Q2: Does this approach better predict program reasoning ability than a simple 3-SAT type benchmark? I admit it seems like it would be, but comparisons to other approaches are sparse.

The key distinction is that Codoku requires joint reasoning about both program structure and execution semantics.

A 3-SAT (or k-SAT) task asks for a Boolean assignment satisfying a propositional formula, analogous to execution prediction: predicting the 3 (or k) inputs to make the formula evaluation/execution true. But Codoku’s unknowns include identifiers, constants, operators, functions, and control-flow targets; their selection together determines execution semantics. A natural constraint-based encoding uses nonlinear finite-domain constraints, rather than a fixed first-order formulation, covering both the unknown structure and the resulting control/data flow.

Answer to W2

W2: The ability of the LLM to use a constraint solver seems like a hole, similar to running a program execution benchmark on an LLM/agent that has access to a code execution tool.

Using constraint solvers (and other tools) is within Codoku’s design.

This differs significantly from simply executing a program, which requires only the ability to call tools such as python. Using a constraint solver for Codoku requires formulating and decomposing the puzzle’s semantic constraints through jointly modeling the structure and the execution. This modeling process is unavoidable and integral to the reasoning capability we evaluate, because incorrect models yield incorrect solutions. This is analogous to a mathematics exam that permits calculators: the student must formulate the problem correctly, while the calculator performs the calculation.

Moreover, using constraint solving directly is practically intractable for Codoku puzzles. Following the reviewers’ suggestions, we built such a full-puzzle SMT baseline using Z3. It solved 0 of 300 puzzles, with one hour per puzzle on one core of an AMD EPYC 7742 64-core processor. Successful agents instead combine local constraint solving with algebraic reasoning, search, repair, and other strategies (Section 4.1).

Thus, Codoku evaluates whether an agent can formulate and solve a puzzle effectively, including through tool use. Its design forces agents to reason about the program’s local state, select tractable subproblems to model, and integrate the solver’s results into a complete solution.

Thanks for the responses, I think a lot of hard work was done to produce this paper, I'm leaving my scores unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.