The weakness would be novelty:
There are other datasets that try and measure LLM reasoning ability on similar sorts of tasks, although I acknowledge it appears PuzzleCodeBench is a much better and more comprehensive dataset.
The RL experiments verify the usefulness of the framework and dataset for LLM finetuning experiments, but don't propose anything novel in the approach.
Overall I think the 47 page appendix is great - a huge resource for a researcher who wants to advance this area. But some conclusions & claims in the paper may need updating if the appendix errors found by the LLM are real. The appendix (47 pages) was too long for me to read thoroughly - so I asked ChatGPT and Claude to review - I include the LLM comments from both below. I burned the tokens - if the claimed errors in the appendix are real please fix and update - but below here is not verified by me - it's all LLM output to review the 47 page appendix. Feel free to ignore.
The benchmark addresses a useful problem, but concrete inconsistencies in the puzzle specifications and evaluation metrics currently undermine confidence in its central claims. I reviewed the paper and appendices, visually checked the problematic examples, and consulted related work.
Summary of the contribution
PuzzleCodeBench evaluates whether vision-language models can turn puzzle images and explicit rules into executable Python solvers. It contains 190 anchor instances and 600 transfer instances across 30 puzzle families. Its standardized observation interface supports separate measurements of image parsing, solver correctness given ground-truth inputs, transfer to other boards, and execution efficiency.
The paper also explores execution-grounded reinforcement learning on Qwen3.5-9B. It reports improved execution accuracy and successful oracle-conditioned solving in nine families excluded from RL training.
The most valuable contribution is the distinction between solving one board and producing a solver that works across boards. This could provide a useful evaluation resource without requiring a new learning algorithm.
Strengths
-
A meaningful evaluation objective. Testing generated programs on held-out boards provides stronger evidence of reusable solver construction than evaluating only the original puzzle answer. Larger and harder transfer instances expose errors that a single successful example can conceal.
-
Useful diagnostic interventions. Executing the same generated program with predicted versus ground-truth observations helps investigate the contribution of parsing errors. The paper also includes image-versus-text and joint-versus-two-step generation comparisons.
-
Broad coverage and substantial experimental work. The benchmark spans placement, shading, number assignment, and connection puzzles. Evaluation includes multiple model families, reasoning settings, and detailed per-puzzle results.
-
Several thoughtful reproducibility provisions. The paper supplies prompts, execution limits, hardware details, RL hyperparameters, and links to artifacts. Appendix C.4 describes repeated profiling trials and a controlled execution stack. Appendix C.2 explicitly addresses solution uniqueness.
-
Promising preliminary training results. Execution accuracy improves from 7.9% to 20.5%, while oracle accuracy improves from 10.5% to 32.1%. Appendix E.1 also includes an SFT comparison and separate unseen-family results; these are useful additions beyond the aggregate improvement.
Major weaknesses and supporting evidence
-
Some published puzzle examples contradict their own rules.
This is my most consequential concern because the benchmark depends on agreement among natural-language specifications, reference solutions, and validators.
Location Concrete inconsistency
Appendix F.2, p. 38: Binairo+ The final answer row is B W B B B W. It contains four black and two white cells, violating the equal-count rule, and three consecutive black cells, violating another explicit rule. This appears in both the rendered answer and the answer literal.
Appendix F.2, p. 50: Stitches Rule 1 requires a stitch between every pair of orthogonally adjacent cells from different regions. However, cells (0,0) and (0,1) belong to different regions, while the first-row clue is zero and the answer places no stitch there. The written rule and the supplied instance are incompatible.
Appendix F.2, p. 51: Tents Rule 1 requires each tent to be adjacent to exactly one tree. The answer places a tent at (3,0), adjacent to trees at both (2,0) and (3,1).
Coordinates above are zero-indexed.
These observations do not establish that the benchmark dataset is corrupted: they could be errors introduced when preparing the appendix. Nevertheless, the authors need to establish which explanation applies. If these rules or solutions are used during evaluation, compliant programs could be penalized or invalid solutions rewarded.
For an execution-centered benchmark, an independent audit of rule–validator–solution consistency is essential.
- The reported efficiency and transfer results conflict with the stated protocol.
Section 2.4 defines a verified solver as one that solves the anchor and all 20 transfer instances, and states that unverified solvers receive zero relative efficiency.
However, Appendix D.7, p. 20 reports for Claude-Opus-4.6:
- Binairo+: 0% verified, but 19.7% relative efficiency and 5.0% win rate.
- Mosaic: 0% verified, but 50.2% relative efficiency and 30.8% win rate.
These cannot follow the stated solver-level verification gate. They could follow a reasonable alternative protocol that rewards individually successful executions, but that protocol must be defined explicitly.
There is a related discrepancy in the anchor gate. Section 2.3 says only oracle-correct anchor solvers proceed to transfer evaluation. Yet Qwen3.5-9B has 0% oracle accuracy and 12.5% transfer accuracy on Minesweeper in Appendix D.7, p. 28. That requires explanation.
These are substantive ambiguities: they change whether efficiency measures reliable reusable solvers, partial correctness, or speed conditional on individual successes.
- The RL evidence supports improvement, but not the full claim of robust reusable algorithms.
The paper’s own benchmark distinguishes:
- Cracked: at least one correct anchor in a family.
- Master: at least one solver that passes the anchor and every transfer case.
The RL section emphasizes improvement in Cracked@30, from 11 to 20, and interprets this as evidence of robust, reusable algorithm design. However, it does not report post-RL Verified@190 or Master@30.
Appendix E.1 strengthens the evidence by reporting 14.2% oracle accuracy and 8.3% execution accuracy on 120 unseen-family anchors. Nevertheless, this establishes some transfer to families unseen during RL, rather than reliable solver generalization within those families.
Additionally:
- Full Reward and Oracle-Only both achieve 20.5% execution accuracy.
- No matched ablation removes only the generalization reward.
- The relationship among the 1,000 additional samples, 110 selected training anchors, reward-evaluation boards, and final benchmark instances needs clearer documentation.
- SFT supervision construction and training settings are insufficiently described for assessing that comparison.
I would retain the RL study as a promising exploratory result, while narrowing the claims or adding the missing transfer evaluation.
- Perception and correctness metrics need explicit handling of equivalent representations.
Observation accuracy is defined by exact equality. Several puzzle observations contain arbitrary region identifiers; several answers contain ordered loop paths or lists of edges.
The paper should explain whether evaluation accepts:
- Identical region partitions with different region IDs.
- The same loop with a different starting point or traversal direction.
- Equivalent edge or stitch collections in different orders.
Solution uniqueness does not resolve these representation ambiguities.
This matters especially because the paper attributes very low region-parsing scores to visual perception. Without documented canonicalization, some failures could reflect serialization conventions. Appendix D.10 usefully discusses genuinely different partitions that preserve an answer, but it does not settle the equivalence-handling question.
- Some broader conclusions exceed what the controls establish.
Several limitations warrant more restrained interpretation:
- Instruction counts are not intrinsic algorithmic complexity. Section 2.4 makes this strong claim, whereas Appendix C.4 correctly acknowledges dependence on the interpreter, processor, and profiling setup. The latter description should be used consistently.
- Twenty transfer cases provide finite test coverage. Passing them is useful evidence, but “Master” should not imply comprehensive family-level correctness.
- Family coverage has unequal sampling opportunities. Anchor counts range from two to twelve per family. “At least one success” metrics are sensitive to this imbalance.
- Uncertainty is not reported. Some families have very few anchors, and modest differences between conditions need paired uncertainty estimates or repeated generations.
- The two-step ablation changes access to rules. Its parsing stage omits puzzle rules, so it does not isolate joint generation alone as the source of better parsing.
- Visual scaling is not tested up to the advertised maximum size. The large transfer boards are supplied as structured oracle observations. The paper should distinguish large-board solver scaling from large-image perception.
Novelty and significance
The combination of visual parsing, reusable solver generation, and cross-instance efficiency evaluation is worthwhile. However, the novelty should be framed around this combination and its implementation.
HumanEval-V already evaluates visual reasoning through executable code generation and test cases. Pencil Puzzle Bench already provides substantial puzzle coverage and deterministic intermediate-state verification. These precedents do not eliminate PuzzleCodeBench’s contribution, but they make broad claims about introducing visual-to-code evaluation or uniquely enabling verifiability too expansive. arxiv.org
A trustworthy, well-documented version of this benchmark could contribute sufficient community value. My recommendation is driven primarily by unresolved validation and measurement issues, rather than the absence of a new algorithm or state-of-the-art result.
Questions for the authors
- Are the Binairo+, Stitches, and Tents discrepancies confined to the appendix? Please provide the actual evaluation rules, stored solutions, and validator outcomes for these examples, together with the results of an independent consistency audit.
- What is the implemented efficiency formula? Please specify the gating and denominator at the solver, instance, and family levels, and explain nonzero efficiency when no solver is verified.
- Are anchor-failing programs evaluated on transfer boards? Please reconcile Section 2.3 with cases having zero oracle anchor accuracy but nonzero transfer accuracy.
- How are equivalent representations normalized? In particular, how are region relabelings, loop rotations/reversals, and unordered edge collections handled?
- What are the post-RL Verified@190 and Master@30 results? Please report them separately for seen and unseen training families, and document disjoint training, reward-evaluation, and final evaluation instances.
- What performs end-to-end output reformatting? Appendix G includes an additional conversion prompt. Which model and settings are used, how often is conversion needed, and how is semantic preservation checked?
Additional feedback to improve the paper
These points are secondary to the recommendation:
- Correct the TopoBench citation in Table 1: the puzzle benchmark is by Maniparambil et al. (2026), whereas the cited Telyatnikov et al. paper concerns topological deep learning. Appendix B already uses the appropriate citation. arxiv.org
- Explain the 53.7% versus 54.2% GPT-5.4@low execution-accuracy discrepancy between Tables 5 and 8.
- Standardize grid-size reporting. Tables 14–15 include 4×4 boards and 26×26 point grids, whereas the main text describes different limits; distinguish cell grids from vertex grids.
- Add a compact protocol table separating E2E accuracy, diagnostic execution accuracy, oracle accuracy, and transfer evaluation.
- Consider rendering-resolution and boundary-contrast ablations to assess whether region failures are specific to the chosen visual presentation.
- Report a rules-and-interface-only solver-generation baseline to measure how much the anchor image contributes to synthesizing a reusable algorithm.
Review: PuzzleCodeBench: A Visual-to-Code Benchmark for Executable Puzzle Solving
Summary
The paper introduces PuzzleCodeBench, a benchmark in which a vision-language model receives an image of a grid logic puzzle plus the rules and must write a complete Python solver. It covers 30 puzzle families in four categories (placement, shading, number, connection), with 190 “anchor” instances (roughly 4×4 to 11×11) used to prompt solver generation and 600 held-out “transfer” instances (up to about 26×26) used to re-run each generated solver.
The evaluation pipeline has two settings:
an end-to-end setting, where the model writes code from the image and rules alone;
a diagnostic setting, where the model must emit a structured OBSERVATION in a given schema plus a solve(observation) function.
The diagnostic setting yields separate metrics:
Perception: Obs. Acc.
Full pipeline: Exec. Acc.
Solver correctness alone: Oracle Acc., computed on the ground-truth observation.
Generalization: Verified@190 and Master@30, where a solver counts as verified only if it passes the anchor and all 20 same-family transfer instances.
Efficiency: Rel. Eff. and Win Rate, based on CPU instruction counts relative to a reference solver.
Eighteen models are evaluated. The main findings:
GPT-5.4@high and Gemini-3.1-Pro@high lead.
Some models (e.g., Claude-Opus-4.6) write good solvers but parse boards poorly.
Region-boundary parsing and path/loop puzzles are the hardest.
Solvers degrade substantially on larger and harder transfer instances.
Finally, the authors apply GSPO with an execution-grounded reward to Qwen3.5-9B on 11 families. They report gains in Oracle Accuracy (10.5% → 32.1%) and Cracked@30 (11 → 20), framed as zero-shot transfer to 9 unseen families.
Strengths
Well-motivated task formulation. Requiring an executable solver rather than a textual answer suits grid puzzles. Solutions are structured objects, correctness is checkable automatically, and one artifact can be tested across many instances. The anchor→transfer protocol is a clean way to ask whether a model produced a procedure or a one-off answer, which text-answer benchmarks cannot measure.
Useful diagnostic decomposition. Separating Obs. Acc. from Oracle Acc. is the paper’s most valuable contribution. It gives a genuinely informative result: models fail for different reasons. Claude-Opus-4.6@max reaches 72.1% Oracle but 19.5% Exec; Doubao-Pro vs. Doubao-Code show opposite profiles. This is the kind of insight a benchmark should provide.
Breadth. Thirty families and a large model sweep, including reasoning-effort ablations (Table 5) and open-weight models (Table 7), give the community a broad snapshot. The per-model, per-family appendix tables are a welcome level of transparency.
Careful efficiency methodology. The efficiency protocol is unusually careful for this area: instruction counts via perf, CPU pinning, warm-up plus trimmed mean, and an explicit acknowledgment that instruction counts are stack-dependent (App. C.4).
Reproducibility. The dataset, code, reference solvers, model outputs and full prompts are released. Solution uniqueness is enforced (App. C.2), which avoids penalizing valid alternative answers.
Training-environment demonstration. The RL section, while preliminary, shows the benchmark’s reward signals are usable for training. The reward ablation (Obs-only vs. Oracle-only vs. full) is a sensible design.
Weaknesses
- Several result patterns suggest possible evaluation or format artifacts rather than model failures
This is my main concern. Some patterns in the appendix tables are hard to explain as genuine reasoning failures.
Slitherlink transfer is 0.0 for every model. This includes models with 100% Oracle Acc. on Slitherlink anchors: Claude-Opus-4.6, Claude-Sonnet-4.6, Claude-Opus-4.7 and GPT-5.4@medium. The transfer pool includes 5×5, 7×7 and 10×10 boards (Table 15), the same sizes as the anchors (Table 14). A solver that is correct on a 5×5 anchor but wrong on every 5×5 transfer instance, for every model, points more toward an answer-canonicalization or format mismatch than toward “global loop reasoning” (L353–357). Examples would be the loop’s start vertex or direction, or the closing vertex. Masyu, also a closed-loop puzzle, reaches 94.3% transfer accuracy for GPT-5.4@high.
Hashi shows the same pattern. Most models reach 100% Oracle on Hashi anchors but 0.0% transfer accuracy (Gemini-3.1-Pro, Gemini-3-Flash, both Claude-Opus-4.6 variants, Doubao, Qwen3.5-Plus), even though the transfer pool includes the anchor sizes 7×7 and 10×10. Only the GPT-5.4 variants succeed. A possible cause is bridge-key ordering in the answer dict, ((r1,c1),(r2,c2)) vs. reversed.
Exec. Acc. exceeds Oracle Acc. on Skyscrapers for many models. Examples are Gemini-3.1-Pro (66.7 vs. 33.3), Kimi-K2.5 (44.4 vs. 11.1), Qwen3.6-Plus (44.4 vs. 22.2) and GPT-5.4@high (88.9 Exec with 33.3 Obs). A solver that succeeds on the model’s own (supposedly wrong) observation but fails on the ground-truth observation suggests the ground-truth observation differs in convention from what the rules or image imply. Obs. Acc. on Skyscrapers is 33.3% for nearly every model, which reinforces this.
Region parsing is near 0% for almost all models (Table 3), and Star Battle (1-star) Obs. Acc. is 0.0 for almost every model despite non-zero Exec. Acc. It is unclear whether region-ID matching is invariant to label permutation. If not, the “region perception bottleneck” (L370–377) may partly be a labeling artifact. The Aquarium case study (App. D.10) shows genuinely different partitions in that instance, but that does not settle the general question.
Claude’s token-level observation accuracy is surprisingly low. On simple token tasks it is 0.23 (Table 3), e.g., 16.7% on Binairo, while most other models score ≈1.0. Grok-4 is 0.06. That gap is large enough that it may reflect schema compliance (e.g., 'W' vs. 'w', None vs. ''), image preprocessing (resizing to 768 px), or an API-side image handling difference, rather than visual perception.
The paper’s headline diagnostic claims rest on these metrics: perception as the dominant bottleneck, region parsing near zero, and loop puzzles unsolved. The patterns above need to be explained or ruled out before the claims can be trusted.
- Limited statistical rigor
Anchor counts per family range from 2 (Nonograms, Star Battle 2-star/shapeless) to 12 (Kakurasu). Per-family metrics therefore move in steps of 8–50 percentage points, and Cracked@30 or Master@30 can flip on a single instance.
All results are single runs (temperature 0, but Kimi at 1.0), with no confidence intervals or seeds.
Many comparisons in the text rest on differences of a few points, for example Table 9 (−3.2 / +2.1 pp) and Table 8. These are likely within noise.
Verified@190 is micro-averaged over anchors, so families with more anchors weigh more. Rel. Eff. and Win Rate are macro-averaged over families. The weighting differs across metrics in the same table without justification.
- Code vs. text comparison (Fig. 1) is narrow and possibly not apples-to-apples
The comparison uses a single model at low reasoning effort (GPT-5.4@low, App. C.1). The 53.7% code figure on PuzzleCodeBench matches the diagnostic setting Exec. Acc. (Table 5), where the reference observation schema is provided. Yet Fig. 2 reports GPT-5.4@low end-to-end code accuracy as 37.9%, which is below the 43.2% text baseline.
The paper should state clearly which setting each number comes from. As presented, the claim that code is a better interface than text depends on that choice. A low-effort text baseline is also a weak comparison for long-chain puzzle solving.
- The end-to-end setting depends on an unspecified LLM reformatter
End-to-end outputs are passed through an LLM “format converter” (App. G) instructed to make “a strong best-effort parse.” The paper does not say which model performs this conversion, how often it intervenes, or whether it could repair or even supply answers. That makes the Fig. 2 headline numbers harder to interpret.
- RL claims are overstated relative to the evidence
“Zero-shot generalization to 9 unseen families” rests on Cracked@30. That metric only requires one anchor in a family to be oracle-correct, and some families have only 2–3 anchors. Table 13 (unseen families only) shows 14.2% Oracle and 8.3% Exec, about 10 of 120 instances, with no pre-RL baseline on those same 19 families. The statement that the model “effectively induces robust, reusable algorithm design rather than merely overfitting” (L469–471) is not supported, especially since no transfer-level metric (Verified@190) is reported after RL.
Data counts don’t add up. “1,000 additional training samples” vs. “10 anchor instances per family” over 11 families. Table 12 lists Qwen3.5-9B-Base, while the text uses the thinking-mode model.
Contamination and baselines. It is unclear whether the 11 training families’ benchmark anchors and transfer instances were excluded from the generated training data, or whether only the generator was shared. The SFT baseline (App. E.1) lacks details on where its targets came from. RL is single-seed.
- Prompt design confounds what is measured
The prompts state that efficiency is “the ultimate metric,” prohibit “naive DFS/BFS,” and require “human-like deduction.” None of these constraints is verified. Strong efficiency language may push models toward fragile, over-engineered propagation code, contributing to the wrong answers seen on transfer. A neutral prompt (correctness first, no stylistic mandates) would help separate model capability from prompt pressure.
- “Extended reasoning” tiers are not compute-matched, and token accounting is unclear
Claude-Opus-4.6@max averages 3,283 output tokens vs. 3,120 at default, while GPT-5.4@high averages 24,228 (Table 10). Either Claude’s thinking tokens are not counted or “max effort” did not take effect.
GPT-5.4 “default” corresponds to reasoning_effort=none (App. C.3), while other providers’ defaults include some reasoning. Cross-vendor tier comparisons, e.g., “Increasing test-time reasoning helps narrow this gap,” should be framed with these caveats.
- Memorization and novelty
These are classic, widely implemented puzzle families; many are in Simon Tatham’s collection, and solvers for them are abundant online. Models may be recalling known solver code rather than synthesizing from rules. The limitations section acknowledges this, but it directly affects the “algorithmic reasoning” interpretation.
On novelty, the comparison in Table 1 omits the large line of work on LLMs generating executable programs or solver encodings for reasoning problems, such as PAL, Program-of-Thoughts and Logic-LM. Code-as-reasoning-interface is established. The contribution is its application to visual grid puzzles with transfer and efficiency evaluation, which should be positioned explicitly.
- Smaller inconsistencies
Citations. TopoBench is cited as Telyatnikov et al. (2025) in Table 1 and the introduction, which is a topological deep learning framework. The appendix correctly cites Maniparambil et al. (2026).
Anchor sizes. The text says anchors span 5×5 to 10×10 (L203), but Table 14 includes 4×4 (Futoshiki, Kakurasu, Renzoku, Skyscrapers) and 11×11 (Shingoki, Slant). The “up to 25×25” claim applies to only some families; Sudoku is 9×9 only and Skyscrapers ≤ 6×6.
Same setting, different numbers. The GPT-5.4@low joint setting gives 54.2% Exec. in Table 8 vs. 53.7% in Table 5.
Tied aggregates. Claude-Opus-4.6 and Claude-Opus-4.6@max have identical aggregate Obs. Acc. (23.7%) and identical Table 3 perception rows, despite different per-family values. Please confirm this is not a copy error.
Untested reference solvers. Win Rate is relative to the authors’ reference solvers, whose quality is not independently validated. If they are weak, win rates are inflated.
The task formulation is valuable, and the Obs/Oracle/Transfer decomposition is a genuine contribution. However:
(i) Several prominent results show patterns more consistent with evaluation or format artifacts than with model failures: Slitherlink and Hashi transfer at 0% even at anchor sizes, and Exec > Oracle on Skyscrapers. The paper’s key diagnostic claims rest on exactly these numbers.
(ii) The small, uneven per-family samples without uncertainty estimates, and the overstated RL generalization claims, weaken the empirical support.
If the authors can show the evaluator is sound on these families (or fix it and update the results), add uncertainty estimates, and temper the RL claims, I would view this as a solid benchmark paper suitable for acceptance.
Supporting arguments
A benchmark’s value lies in the trustworthiness of its measurements. The paper’s distinctive claims are of three kinds:
which capability is the bottleneck (perception, especially regions);
which puzzle types defeat all models (Slitherlink, Yin-Yang, loops);
how much solvers generalize (Verified@190).
Each of these depends on the per-family numbers flagged in Weakness 1. The anomalies are concentrated in specific families and cut across all models, which is the signature of a harness issue rather than a capability gap. Without resolution, practitioners could draw wrong conclusions, for example that loop reasoning is uniformly unsolved when the issue may be answer canonicalization.
The RL contribution is explicitly framed as preliminary. That is fine, but the stated conclusions should match that framing.
Questions for the authors
Slitherlink and Hashi. Why is transfer accuracy 0.0 for models with 100% anchor Oracle Acc., including on transfer instances of the same size as the anchors? How are loop answers (start vertex, direction, closure) and Hashi bridge keys (endpoint order) canonicalized before comparison? Could you report transfer accuracy restricted to anchor-sized instances?
Skyscrapers. How can Exec. Acc. exceed Oracle Acc. (e.g., Kimi 44.4 vs. 11.1)? Does the ground-truth observation use a convention (clue ordering, missing-clue encoding) that differs from what the image and rules imply?
Observation matching. Is Obs. Acc. invariant to region-ID relabeling? Does it normalize equivalent encodings (None vs. '', case)? What fraction of observation errors are schema or formatting issues vs. genuine misreadings, especially for Claude and Grok on token tasks?
End-to-end reformatter. Which model performs the format conversion? How often does it change or recover answers, and have you audited it for solving rather than reformatting?
Fig. 1. Is the 53.7% code number the diagnostic setting? How does code compare to text under the true end-to-end setting (37.9% vs. 43.2% for GPT-5.4@low)? Can the comparison be extended to high reasoning effort and at least one more model?
Uncertainty. Can you report confidence intervals (e.g., bootstrap over instances) and, for a few models, variance across multiple samples or seeds?
RL.
What is the pre-RL baseline on the 19 unseen families in Table 13?
What are post-RL transfer metrics (Verified@190, Master@30)?
How exactly were training instances separated from benchmark anchors and transfer pools?
How do “1,000 samples” and “10 anchors per family” reconcile, and was the base model Qwen3.5-9B-Base or the instruct/thinking model?
Token counts. Are Claude’s thinking tokens included in Table 10? Was the effort: max setting verified to take effect?
Reference solvers. How strong are the reference solvers (e.g., compared against established puzzle solvers)? Are they implemented in pure Python like the generated solvers, so instruction-count ratios are comparable?
Prompt sensitivity. How sensitive are the results to the efficiency- and “human-like deduction”-heavy prompt? A neutral-prompt ablation on a subset would help.
Additional feedback (to help improve the paper, not part of my decision)
Consider balancing the anchor set (equal anchors per family) or reporting both micro and macro averages consistently across all metrics.
Report the mapping from each family to the perception types (numeric, token, region, symbol) used in Table 3.
Add a short failure taxonomy based on manually inspected transfer failures (missing constraint vs. weak propagation vs. timeout). This would back the claims in L405–415 more directly than aggregate counts.
Position the work against code-as-reasoning approaches (PAL, Program-of-Thoughts, Logic-LM) and visual program-synthesis benchmarks in the related-work discussion and Table 1.
A contamination probe would strengthen the “algorithm synthesis” framing, e.g., rule variants or novel families not found online.
Fix the TopoBench citation, the anchor-size statement, and the Table 5/Table 8 discrepancy.
Fig. 2 lists a subset of models with slightly different names from Table 2 (“Claude-Opus-4.6” vs. “@max”). Aligning them would avoid confusion.
Polish: “a complete, stateful, and automatically verifiable solving procedures” (L42).
Response to Reviewer vSrU (Part 1/2)
We sincerely appreciate the reviewer's time and effort in reviewing our paper and providing constructive feedback. Our responses and clarifications are provided below.
Thank you for raising this concern. The 30 families cover diverse rule systems, including binary balancing (Binairo), Latin-square constraints (Sudoku), inequalities (Futoshiki), neighborhood counting (Minesweeper), run-length constraints (Nonograms), region quotas (Star Battle), line-of-sight constraints (Kurodoko), polyomino placement (LITS), bridge connectivity (Hashi), and loop construction (Slitherlink). Within each family, anchor and transfer instances vary in board configurations and cover different difficulty levels (easy, normal, and hard), with difficulty calibrated using the reference solver's execution time and search depth. Family-specific grid sizes range from 4×4 to 25×25 across the benchmark.
Inter-family correlation. Following LongBench [1] (Appendix D), we compute pairwise Spearman correlations (ρ) between family performance vectors across all 30 families, using the 16 main model configurations in Table 2 (excluding the two smaller Qwen baselines), yielding 435 possible family pairs. Pairs with constant scores are excluded. We report median ρ and proportions below 0.2 or above 0.8 as descriptive thresholds.
Only 0.7–3.7% of valid family pairs exhibit high positive correlations (ρ > 0.8), with lower median correlations for Oracle and Transfer accuracy. These results indicate heterogeneous family-specific performance patterns rather than widespread redundancy.
Subset stability. Over 2,000 draws per strategy, we evaluate (1) family subsampling, selecting 10/15/20 families proportionally across four categories while retaining all anchors; and (2) anchor subsampling, retaining all 30 families with 1 or up to 3 randomly selected anchors per family, preserving original family weights. Using the original aggregation protocol, we compare subset rankings (Spearman ρ) and scores (MAE, percentage points) against the full benchmark, averaging over draws.
Although smaller subsets largely preserve overall model rankings, they introduce non-negligible errors in absolute performance estimates. Notably, at comparable anchor budgets, retaining all 30 families with fewer anchors yields lower score errors than selecting only 15 families. These findings highlight the value of broad family coverage for reliable performance measurement and family-specific diagnostics, beyond approximating overall model rankings.
Carefully curated subsets. We further evaluated greedy selection of 10/15 families over 200 model-level splits (8 models for selection and 8 held out for evaluation), using 50 category-matched random subsets per split as baselines. Curated subsets showed no consistent advantage over stratified random subsets on held-out models (ΔSpearman, curated minus random, for Exec./Oracle/Transfer: +0.001/−0.010/−0.010 for 10 families; +0.003/−0.014/+0.004 for 15 families).
Overall, compact subsets can support low-cost screening, while the full benchmark provides more faithful absolute performance measurements and comprehensive family-level diagnostics. We have added detailed experimental settings and complete results to Appendix D.11.
[1] Bai et al. "LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024.