acceptodds
Under review as a conference paper at ICLR 2027

PuzzleCodeBench: A Visual-to-Code Benchmark for Executable Puzzle Solving

Abstract

Visual puzzles offer a challenging testbed for vision-language models (VLMs), requiring visual parsing, rule grounding, structured state tracking, and verifiable solution search. However, existing benchmarks cover limited scales and complexities, and typically evaluate models via natural-language answers. Such non-executable, instance-specific responses make it difficult to verify whether models learn reusable solving strategies, limiting the evaluation of scalable generalization and efficiency. To address these limitations, we introduce **PuzzleCodeBench**, which reframes puzzle solving under a visual-to-code paradigm: given a puzzle image and rules, models must parse the visual board and generate a complete Python solver that computes the solution. The benchmark comprises 790 instances across 30 grid-based logic puzzle families and grids up to 2525. Based on this collection, we develop an execution-centered pipeline with a standardized observation interface to evaluate models along four axes: visual perception, solver correctness, cross-instance generalization, and algorithmic efficiency. Experiments across frontier VLMs show that executable puzzle solving remains challenging, with model-specific bottlenecks in visual parsing and solver synthesis. Meanwhile, even strong models show substantial performance drops on larger and harder held-out instances. We further apply execution-grounded reinforcement learning to Qwen3.5-9B on 11 puzzle families, achieving a 2.5 improvement in execution accuracy and zero-shot generalization to 9 unseen families. These results highlight **PuzzleCodeBench** as both a diagnostic benchmark and a training environment for robust code-based puzzle solving.

Primary area:
datasets and benchmarks
Keywords:
Puzzle, Benchmark, VLM, Code

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vSrUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The author introduces PuzzleCodeBench, a benchmark/dataset includes grid-based puzzles from different domain. RL from these puzzles can improve the ability of the VLM/LLM on code-based puzzle solving.

Critical strengths
  • The paper give a good source for benchmark VLM in puzzle solving capability.

  • The benchmark is also a good environment for reinforcement learning.

Critical weaknesses

First, I want to mention that I don't like benchmark paper, although they are quite important and useful... Hence, I am not good at how good a benchmark paper is... AC please pay less attention on my opinions.

  • First, a critical weakness of the proposed benchmark lies in the unverified independence of its numerous subtasks. While comprehensive coverage is valuable, the authors do not sufficiently analyze the correlation between model performances across these distinct subdomains. If results on several subtasks are highly correlated, it suggests substantial redundancy within the evaluation suite. Consequently, it remains unclear whether the full battery of tests is genuinely necessary, or if a significantly smaller, carefully curated subset of tasks would produce the exact same model rankings with a fraction of the computational overhead. Establishing the statistical independence of these subdomains is essential to justify the benchmark's scale and to ensure it measures genuinely diverse capabilities rather than repeatedly testing a single underlying competency.

  • Second, the manuscript lacks a compelling articulation of its unique value proposition relative to the densely populated landscape of existing Vision-Language Model (VLM) benchmarks. While the authors introduce a new evaluation suite, they do not explicitly clarify the fundamental differentiators that set this benchmark apart from established predecessors. A robust benchmark must do more than aggregate new queries; it must address a specific, previously unmeasured axis of evaluation—such as novel reasoning paradigms, unique multimodal interactions, or specific failure modes prevalent in current state-of-the-art models. Without a direct and rigorous comparison illustrating exactly what this benchmark captures that prior VLM evaluations cannot, its overarching importance, necessity, and potential impact on guiding future model development remain ambiguous.

Questions for authors
  • Could you provide the cost (both time and token usage) of the evaluation for some models?

  • If we use this environments as training environment for bench, can the reasoning capability transfer into other general visual reasoning domain rather than just puzzle domains?

4 comments
AuthorsOpenReview ↗

Response to Reviewer vSrU (Part 1/2)

We sincerely appreciate the reviewer's time and effort in reviewing our paper and providing constructive feedback. Our responses and clarifications are provided below.

W1: Inter-family Correlation and Benchmark Redundancy

Thank you for raising this concern. The 30 families cover diverse rule systems, including binary balancing (Binairo), Latin-square constraints (Sudoku), inequalities (Futoshiki), neighborhood counting (Minesweeper), run-length constraints (Nonograms), region quotas (Star Battle), line-of-sight constraints (Kurodoko), polyomino placement (LITS), bridge connectivity (Hashi), and loop construction (Slitherlink). Within each family, anchor and transfer instances vary in board configurations and cover different difficulty levels (easy, normal, and hard), with difficulty calibrated using the reference solver's execution time and search depth. Family-specific grid sizes range from 4×4 to 25×25 across the benchmark.

Inter-family correlation. Following LongBench [1] (Appendix D), we compute pairwise Spearman correlations (ρ) between family performance vectors across all 30 families, using the 16 main model configurations in Table 2 (excluding the two smaller Qwen baselines), yielding 435 possible family pairs. Pairs with constant scores are excluded. We report median ρ and proportions below 0.2 or above 0.8 as descriptive thresholds.

MetricMedian ρρ < 0.2ρ > 0.8
Exec. Acc.0.5048.4%3.7%
Oracle Acc.0.39021.6%0.7%
Transfer Acc.0.39523.6%1.7%

Only 0.7–3.7% of valid family pairs exhibit high positive correlations (ρ > 0.8), with lower median correlations for Oracle and Transfer accuracy. These results indicate heterogeneous family-specific performance patterns rather than widespread redundancy.

Subset stability. Over 2,000 draws per strategy, we evaluate (1) family subsampling, selecting 10/15/20 families proportionally across four categories while retaining all anchors; and (2) anchor subsampling, retaining all 30 families with 1 or up to 3 randomly selected anchors per family, preserving original family weights. Using the original aggregation protocol, we compare subset rankings (Spearman ρ) and scores (MAE, percentage points) against the full benchmark, averaging over draws.

SubsetRanking ρ (Exec./Oracle/Transfer)Score MAE (pp, Exec./Oracle/Transfer)
10 families0.948 / 0.916 / 0.8866.17 / 6.02 / 6.26
15 families0.971 / 0.941 / 0.9164.42 / 4.25 / 4.46
20 families0.983 / 0.965 / 0.9463.12 / 3.00 / 3.13
1 anchor/family0.920 / 0.888 / –5.25 / 5.93 / –
Up to 3 anchors/family0.975 / 0.953 / –2.51 / 2.83 / –

Although smaller subsets largely preserve overall model rankings, they introduce non-negligible errors in absolute performance estimates. Notably, at comparable anchor budgets, retaining all 30 families with fewer anchors yields lower score errors than selecting only 15 families. These findings highlight the value of broad family coverage for reliable performance measurement and family-specific diagnostics, beyond approximating overall model rankings.

Carefully curated subsets. We further evaluated greedy selection of 10/15 families over 200 model-level splits (8 models for selection and 8 held out for evaluation), using 50 category-matched random subsets per split as baselines. Curated subsets showed no consistent advantage over stratified random subsets on held-out models (ΔSpearman, curated minus random, for Exec./Oracle/Transfer: +0.001/−0.010/−0.010 for 10 families; +0.003/−0.014/+0.004 for 15 families).

Overall, compact subsets can support low-cost screening, while the full benchmark provides more faithful absolute performance measurements and comprehensive family-level diagnostics. We have added detailed experimental settings and complete results to Appendix D.11.

[1] Bai et al. "LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024.

AuthorsOpenReview ↗

Response to Reviewer vSrU (Part 2/2)

W2: Unique Value of PuzzleCodeBench

Thank you for this comment. Table 1 shows that PuzzleCodeBench uniquely combines visual inputs and executable solver outputs among the compared benchmarks, covering the most grid-based puzzle families (30). More importantly, it requires VLMs to translate visual puzzles and rules into reusable executable solving procedures, rather than merely generating instance-level answers.

This formulation enables systematic evaluation of capabilities beyond conventional puzzle-solving benchmarks. By executing each generated solver on its anchor and 20 held-out instances of the same family, we evaluate not only visual perception and solver correctness, but also cross-instance generalization and computational efficiency. This assesses whether models can construct algorithms that correctly implement puzzle rules and generalize across different instances, rather than merely solving individual puzzles.

Furthermore, our evaluation reveals distinct model-specific bottlenecks (e.g., Claude-Opus-4.6@max achieves 72.1% Oracle Acc. but only 19.5% Exec. Acc., Table 2) and highlights how visual input can influence solver synthesis, with effects varying across models and puzzle families (Appendix D.6). These findings provide actionable insights for improving visual grounding and algorithmic reasoning in VLMs.

Q1: Evaluation Cost (Time and Tokens)

Thank you for this question. Average output token usage across 190 anchor instances is reported in Appendix D.9 (Table 10). Notably, GPT-5.4@high and Gemini-3.1-Pro@high, the two best-performing models, both consume approximately 24k tokens per instance.

We additionally measured the average response time of four representative models:

ModelAverage Response Time (s)
GPT-5.4@high366.86
GPT-5.4122.37
Gemini-3.1-Pro@high196.91
Qwen3.6-Plus235.81

We note that wall-clock time is affected by provider-side infrastructure and API/network conditions, and thus is not an intrinsic measure of model efficiency.

Q2: Cross-Domain Visual Reasoning Transfer

Thank you for this question. To investigate transfer beyond puzzle domains, we evaluate on HumanEval-V [1], a benchmark of 253 diagram-based visual reasoning and code generation tasks. We compare the original Qwen3.5-9B@Thinking with our RL-trained Qwen3.5-9B (Section 4), trained on 11 puzzle families, using the official V2C prompt and evaluator with identical decoding settings (32k max tokens, temperature 0.6, n=1). Pass@1 improves from 58.9% to 62.5% (+3.6 percentage points), suggesting modest positive transfer to non-puzzle visual reasoning and code generation tasks. We have added this cross-domain evaluation and its results to Section 4.

[1] Zhang et al. "HumanEval-V: Systematic Evaluation of Visual Reasoning in Large Multimodal Models for Code Generation." ACM Transactions on Software Engineering and Methodology, 2026.

Thank you for your response which solves my concern! I will raise the score. Hope the benchmark will be public after accepted.

Thank you very much for your quick response and willingness to raise your score. We sincerely appreciate your positive feedback and encouragement!

Reviewer_fMxEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces PuzzleCodeBench, a benchmark for evaluating vision-language models on their ability to develop reusable solving strategies in the form of code generation given a sample visual logic puzzle. The benchmark contains 790 puzzle instances across 30 puzzle families, spanning four categories of reasoning tasks. Unlike previous puzzle benchmarks that evaluate whether a model can reason through it to reach a final answer, PuzzleCodeBench focuses on evaluating their program synthesis capabilities on four aspects: visual perception, solver correctness, cross-instance generalization, and computational efficiency. Experiments across frontier VLMs reveal substantial limitations in visual parsing and solver generalization. The authors also demonstrate that using reinforcement learning on the synthesized solver program's execution feedback improves Qwen3.5-9B's performance from 7.9% to 20.5%, and exhibits transfer to unseen puzzle familities.

Critical strengths
  • I think the benchmark is well motivated, as constructing reusable solver programs from a puzzle instance and instructions is an important reasoning capability. Although the idea of using program synthesis to solve visual puzzles is not new, I like how the authors constructed the benchmark, providing decomposition on the model performance on the four aspects of puzzle understanding and solver synthesis.
  • The benchmark covers a large set of 30 puzzle families with varying grid sizes and difficulty, and the evaluation of frontier VLMs show important limitations in their visual reasoning and program synthesis.
  • The authors demonstrate that reinforcement learning on execution feedback can improve solver synthesis in a relatively small model, including some transfer to unseen puzzle families.
Critical weaknesses
  • there are limitations to the authors' RL experiments. For example, the authors only Cracked@30 metric rather than the Verified@190 and Master@30 metrics in the main table, so it is not clear whether RL actually improves the model's cross-family generalization.
  • while the author prompts the evaluated models to not use external solvers, it is not immediately clear how that is enforced. There doesn't seem to be guardrails that prevent a "cheating" model to use these external solvers.
Questions for authors
  • could the authors evaluate the RL-trained model using Verified@190 and Master@30 metrics, particularly on the puzzle families excluded from training?
  • could the authors also provide more information on how they enforce that models cannot use external solvers? Are network access disabled, and are generated programs automatically checked for prohibited dependencies?
3 comments
AuthorsOpenReview ↗

Response to Reviewer fMxE

We thank the reviewer for the thoughtful feedback and valuable suggestions. We provide the following detailed responses to address the reviewer’s comments and concerns.

W1 & Q1: RL Solver Generalization

Thank you for this great suggestion. We agree that Cracked@30 alone cannot establish solver generalization. We therefore additionally evaluate Verified and Master metrics before and after RL, separating the 30 puzzle families into 11 training families and 19 held-out families. The evaluation covers 70 and 120 benchmark anchor instances, respectively, with each generated solver tested on 20 same-family transfer instances.

SplitModelCrackedMasterVerified
Training (11 families, 70 anchors)Before RL11611/70 (15.7%)
After RL11826/70 (37.1%)
Held-out (19 families, 120 anchors)Before RL000/120 (0.0%)
After RL925/120 (4.2%)
Overall (30 families, 190 anchors)Before RL11611/190 (5.8%)
After RL201031/190 (16.3%)

Notably, on the 19 held-out families, RL improves Cracked from 0 to 9, Master from 0 to 2, and Verified from 0 to 5, with the two mastered families being Minesweeper and Binairo. Therefore, these results demonstrate that execution-grounded RL improves both within-family solver generalization and cross-family transfer, although robust generalization to unseen families remains challenging. We have incorporated these results into the revised manuscript (Section 4 and Appendix E.2).

W2 & Q2: External Solver Enforcement

Thank you for raising this concern. Beyond prompt-level restrictions, all generated programs are executed as isolated processes in the clean, fixed Linux container used for evaluation, which contains no third-party solver packages and has network access disabled. We also audited outputs from four representative models (Gemini-3.1-Pro@high, GPT-5.4@low, Claude-Opus-4.6, and Qwen3.6-Plus), using GPT-5.6-luna as an LLM judge followed by human verification. No prohibited solver usage, suspicious dependencies, or circumvention attempts were identified in these audited outputs.

Reviewer_fMxEOpenReview ↗

Thank you for the response

Thank you for the response! I see that RL does produce some cross-family transfer on this benchmark, so I have no remaining concerns.

My score was already leaning accept, so I will maintain that score. I think a stronger version of this paper could further explore the relationship between a model's puzzle-solving capabilities on this benchmark and its performance on downstream tasks, such as math and coding. That said, I believe the paper in its current form is already solid and provides a valuable contribution to the community.

We sincerely thank you for your encouraging feedback and thoughtful suggestions. Exploring the relationship between puzzle-solving capabilities and downstream math and coding performance is an interesting direction. Following Reviewer vSrU's suggestion, we have taken an initial step by evaluating cross-domain transfer on HumanEval-V, where our RL-trained Qwen3.5-9B improves Pass@1 from 58.9% to 62.5% (Section 4, Cross-domain Generalization). We will further explore this direction in future work. We sincerely appreciate your valuable suggestions and continued support.

Reviewer_AoFoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper PuzzleCodeBench creates a large dataset of many types of visual problems that come with rules to test LLM on solving visual reasoning problems. The paper proposes that solutions should be executable code that can generalize to solving other similar problems, so rather than solve a game instance of Sudoku, come up with a program to solve all Sudoku game instances. The paper has a framework that allows the LLM to solve the problem end-to-end, but also can be run to separate out the task of converting the image to a problem state, and the problem of generating correct code to convert the problem state to a full solution, so the visual mistakes can be isolated from the logical coding mistakes. RL experiments were done to verify the dataset & framework can be used successfully to finetune an LLM to do better.

Critical strengths

An impressive amount of hard work was done preparing the paper, the dataset, framework and large number of evaluations are non-trivial. The dataset created for the paper would be useful for other researchers that want to test and/or finetune&post-train their VLM model on visual reasoning problems, with a metric that really ensures the LLM has found a general solution. The paper frames the problem of understanding how to solve a visual problem in a more thorough way - don't just solve specific instances of a problem but come up with an algorithm that can solve all instances of a problem. The paper tries to tease apart the errors from visual understanding of image, from the logical reasoning steps. The appendix and open source code will provide a great on-ramp for other researchers who want to push in this direction.

Critical weaknesses

The weakness would be novelty: There are other datasets that try and measure LLM reasoning ability on similar sorts of tasks, although I acknowledge it appears PuzzleCodeBench is a much better and more comprehensive dataset. The RL experiments verify the usefulness of the framework and dataset for LLM finetuning experiments, but don't propose anything novel in the approach.

Overall I think the 47 page appendix is great - a huge resource for a researcher who wants to advance this area. But some conclusions & claims in the paper may need updating if the appendix errors found by the LLM are real. The appendix (47 pages) was too long for me to read thoroughly - so I asked ChatGPT and Claude to review - I include the LLM comments from both below. I burned the tokens - if the claimed errors in the appendix are real please fix and update - but below here is not verified by me - it's all LLM output to review the 47 page appendix. Feel free to ignore.


The benchmark addresses a useful problem, but concrete inconsistencies in the puzzle specifications and evaluation metrics currently undermine confidence in its central claims. I reviewed the paper and appendices, visually checked the problematic examples, and consulted related work.

Summary of the contribution

PuzzleCodeBench evaluates whether vision-language models can turn puzzle images and explicit rules into executable Python solvers. It contains 190 anchor instances and 600 transfer instances across 30 puzzle families. Its standardized observation interface supports separate measurements of image parsing, solver correctness given ground-truth inputs, transfer to other boards, and execution efficiency.

The paper also explores execution-grounded reinforcement learning on Qwen3.5-9B. It reports improved execution accuracy and successful oracle-conditioned solving in nine families excluded from RL training.

The most valuable contribution is the distinction between solving one board and producing a solver that works across boards. This could provide a useful evaluation resource without requiring a new learning algorithm.

Strengths

  1. A meaningful evaluation objective. Testing generated programs on held-out boards provides stronger evidence of reusable solver construction than evaluating only the original puzzle answer. Larger and harder transfer instances expose errors that a single successful example can conceal.

  2. Useful diagnostic interventions. Executing the same generated program with predicted versus ground-truth observations helps investigate the contribution of parsing errors. The paper also includes image-versus-text and joint-versus-two-step generation comparisons.

  3. Broad coverage and substantial experimental work. The benchmark spans placement, shading, number assignment, and connection puzzles. Evaluation includes multiple model families, reasoning settings, and detailed per-puzzle results.

  4. Several thoughtful reproducibility provisions. The paper supplies prompts, execution limits, hardware details, RL hyperparameters, and links to artifacts. Appendix C.4 describes repeated profiling trials and a controlled execution stack. Appendix C.2 explicitly addresses solution uniqueness.

  5. Promising preliminary training results. Execution accuracy improves from 7.9% to 20.5%, while oracle accuracy improves from 10.5% to 32.1%. Appendix E.1 also includes an SFT comparison and separate unseen-family results; these are useful additions beyond the aggregate improvement. Major weaknesses and supporting evidence

  6. Some published puzzle examples contradict their own rules.

This is my most consequential concern because the benchmark depends on agreement among natural-language specifications, reference solutions, and validators. Location Concrete inconsistency Appendix F.2, p. 38: Binairo+ The final answer row is B W B B B W. It contains four black and two white cells, violating the equal-count rule, and three consecutive black cells, violating another explicit rule. This appears in both the rendered answer and the answer literal. Appendix F.2, p. 50: Stitches Rule 1 requires a stitch between every pair of orthogonally adjacent cells from different regions. However, cells (0,0) and (0,1) belong to different regions, while the first-row clue is zero and the answer places no stitch there. The written rule and the supplied instance are incompatible. Appendix F.2, p. 51: Tents Rule 1 requires each tent to be adjacent to exactly one tree. The answer places a tent at (3,0), adjacent to trees at both (2,0) and (3,1).

Coordinates above are zero-indexed.

These observations do not establish that the benchmark dataset is corrupted: they could be errors introduced when preparing the appendix. Nevertheless, the authors need to establish which explanation applies. If these rules or solutions are used during evaluation, compliant programs could be penalized or invalid solutions rewarded.

For an execution-centered benchmark, an independent audit of rule–validator–solution consistency is essential.

  1. The reported efficiency and transfer results conflict with the stated protocol.

Section 2.4 defines a verified solver as one that solves the anchor and all 20 transfer instances, and states that unverified solvers receive zero relative efficiency.

However, Appendix D.7, p. 20 reports for Claude-Opus-4.6:

  • Binairo+: 0% verified, but 19.7% relative efficiency and 5.0% win rate.
  • Mosaic: 0% verified, but 50.2% relative efficiency and 30.8% win rate. These cannot follow the stated solver-level verification gate. They could follow a reasonable alternative protocol that rewards individually successful executions, but that protocol must be defined explicitly.

There is a related discrepancy in the anchor gate. Section 2.3 says only oracle-correct anchor solvers proceed to transfer evaluation. Yet Qwen3.5-9B has 0% oracle accuracy and 12.5% transfer accuracy on Minesweeper in Appendix D.7, p. 28. That requires explanation.

These are substantive ambiguities: they change whether efficiency measures reliable reusable solvers, partial correctness, or speed conditional on individual successes.

  1. The RL evidence supports improvement, but not the full claim of robust reusable algorithms.

The paper’s own benchmark distinguishes:

  • Cracked: at least one correct anchor in a family.
  • Master: at least one solver that passes the anchor and every transfer case. The RL section emphasizes improvement in Cracked@30, from 11 to 20, and interprets this as evidence of robust, reusable algorithm design. However, it does not report post-RL Verified@190 or Master@30.

Appendix E.1 strengthens the evidence by reporting 14.2% oracle accuracy and 8.3% execution accuracy on 120 unseen-family anchors. Nevertheless, this establishes some transfer to families unseen during RL, rather than reliable solver generalization within those families.

Additionally:

  • Full Reward and Oracle-Only both achieve 20.5% execution accuracy.
  • No matched ablation removes only the generalization reward.
  • The relationship among the 1,000 additional samples, 110 selected training anchors, reward-evaluation boards, and final benchmark instances needs clearer documentation.
  • SFT supervision construction and training settings are insufficiently described for assessing that comparison. I would retain the RL study as a promising exploratory result, while narrowing the claims or adding the missing transfer evaluation.
  1. Perception and correctness metrics need explicit handling of equivalent representations.

Observation accuracy is defined by exact equality. Several puzzle observations contain arbitrary region identifiers; several answers contain ordered loop paths or lists of edges.

The paper should explain whether evaluation accepts:

  • Identical region partitions with different region IDs.
  • The same loop with a different starting point or traversal direction.
  • Equivalent edge or stitch collections in different orders. Solution uniqueness does not resolve these representation ambiguities.

This matters especially because the paper attributes very low region-parsing scores to visual perception. Without documented canonicalization, some failures could reflect serialization conventions. Appendix D.10 usefully discusses genuinely different partitions that preserve an answer, but it does not settle the equivalence-handling question.

  1. Some broader conclusions exceed what the controls establish.

Several limitations warrant more restrained interpretation:

  • Instruction counts are not intrinsic algorithmic complexity. Section 2.4 makes this strong claim, whereas Appendix C.4 correctly acknowledges dependence on the interpreter, processor, and profiling setup. The latter description should be used consistently.
  • Twenty transfer cases provide finite test coverage. Passing them is useful evidence, but “Master” should not imply comprehensive family-level correctness.
  • Family coverage has unequal sampling opportunities. Anchor counts range from two to twelve per family. “At least one success” metrics are sensitive to this imbalance.
  • Uncertainty is not reported. Some families have very few anchors, and modest differences between conditions need paired uncertainty estimates or repeated generations.
  • The two-step ablation changes access to rules. Its parsing stage omits puzzle rules, so it does not isolate joint generation alone as the source of better parsing.
  • Visual scaling is not tested up to the advertised maximum size. The large transfer boards are supplied as structured oracle observations. The paper should distinguish large-board solver scaling from large-image perception. Novelty and significance

The combination of visual parsing, reusable solver generation, and cross-instance efficiency evaluation is worthwhile. However, the novelty should be framed around this combination and its implementation.

HumanEval-V already evaluates visual reasoning through executable code generation and test cases. Pencil Puzzle Bench already provides substantial puzzle coverage and deterministic intermediate-state verification. These precedents do not eliminate PuzzleCodeBench’s contribution, but they make broad claims about introducing visual-to-code evaluation or uniquely enabling verifiability too expansive. arxiv.org

A trustworthy, well-documented version of this benchmark could contribute sufficient community value. My recommendation is driven primarily by unresolved validation and measurement issues, rather than the absence of a new algorithm or state-of-the-art result.

Questions for the authors

  1. Are the Binairo+, Stitches, and Tents discrepancies confined to the appendix? Please provide the actual evaluation rules, stored solutions, and validator outcomes for these examples, together with the results of an independent consistency audit.
  2. What is the implemented efficiency formula? Please specify the gating and denominator at the solver, instance, and family levels, and explain nonzero efficiency when no solver is verified.
  3. Are anchor-failing programs evaluated on transfer boards? Please reconcile Section 2.3 with cases having zero oracle anchor accuracy but nonzero transfer accuracy.
  4. How are equivalent representations normalized? In particular, how are region relabelings, loop rotations/reversals, and unordered edge collections handled?
  5. What are the post-RL Verified@190 and Master@30 results? Please report them separately for seen and unseen training families, and document disjoint training, reward-evaluation, and final evaluation instances.
  6. What performs end-to-end output reformatting? Appendix G includes an additional conversion prompt. Which model and settings are used, how often is conversion needed, and how is semantic preservation checked? Additional feedback to improve the paper

These points are secondary to the recommendation:

  • Correct the TopoBench citation in Table 1: the puzzle benchmark is by Maniparambil et al. (2026), whereas the cited Telyatnikov et al. paper concerns topological deep learning. Appendix B already uses the appropriate citation. arxiv.org
  • Explain the 53.7% versus 54.2% GPT-5.4@low execution-accuracy discrepancy between Tables 5 and 8.
  • Standardize grid-size reporting. Tables 14–15 include 4×4 boards and 26×26 point grids, whereas the main text describes different limits; distinguish cell grids from vertex grids.
  • Add a compact protocol table separating E2E accuracy, diagnostic execution accuracy, oracle accuracy, and transfer evaluation.
  • Consider rendering-resolution and boundary-contrast ablations to assess whether region failures are specific to the chosen visual presentation.
  • Report a rules-and-interface-only solver-generation baseline to measure how much the anchor image contributes to synthesizing a reusable algorithm.

Review: PuzzleCodeBench: A Visual-to-Code Benchmark for Executable Puzzle Solving Summary

The paper introduces PuzzleCodeBench, a benchmark in which a vision-language model receives an image of a grid logic puzzle plus the rules and must write a complete Python solver. It covers 30 puzzle families in four categories (placement, shading, number, connection), with 190 “anchor” instances (roughly 4×4 to 11×11) used to prompt solver generation and 600 held-out “transfer” instances (up to about 26×26) used to re-run each generated solver.

The evaluation pipeline has two settings:

an end-to-end setting, where the model writes code from the image and rules alone; a diagnostic setting, where the model must emit a structured OBSERVATION in a given schema plus a solve(observation) function.

The diagnostic setting yields separate metrics:

Perception: Obs. Acc. Full pipeline: Exec. Acc. Solver correctness alone: Oracle Acc., computed on the ground-truth observation. Generalization: Verified@190 and Master@30, where a solver counts as verified only if it passes the anchor and all 20 same-family transfer instances. Efficiency: Rel. Eff. and Win Rate, based on CPU instruction counts relative to a reference solver.

Eighteen models are evaluated. The main findings:

GPT-5.4@high and Gemini-3.1-Pro@high lead. Some models (e.g., Claude-Opus-4.6) write good solvers but parse boards poorly. Region-boundary parsing and path/loop puzzles are the hardest. Solvers degrade substantially on larger and harder transfer instances.

Finally, the authors apply GSPO with an execution-grounded reward to Qwen3.5-9B on 11 families. They report gains in Oracle Accuracy (10.5% → 32.1%) and Cracked@30 (11 → 20), framed as zero-shot transfer to 9 unseen families.

Strengths Well-motivated task formulation. Requiring an executable solver rather than a textual answer suits grid puzzles. Solutions are structured objects, correctness is checkable automatically, and one artifact can be tested across many instances. The anchor→transfer protocol is a clean way to ask whether a model produced a procedure or a one-off answer, which text-answer benchmarks cannot measure. Useful diagnostic decomposition. Separating Obs. Acc. from Oracle Acc. is the paper’s most valuable contribution. It gives a genuinely informative result: models fail for different reasons. Claude-Opus-4.6@max reaches 72.1% Oracle but 19.5% Exec; Doubao-Pro vs. Doubao-Code show opposite profiles. This is the kind of insight a benchmark should provide. Breadth. Thirty families and a large model sweep, including reasoning-effort ablations (Table 5) and open-weight models (Table 7), give the community a broad snapshot. The per-model, per-family appendix tables are a welcome level of transparency. Careful efficiency methodology. The efficiency protocol is unusually careful for this area: instruction counts via perf, CPU pinning, warm-up plus trimmed mean, and an explicit acknowledgment that instruction counts are stack-dependent (App. C.4). Reproducibility. The dataset, code, reference solvers, model outputs and full prompts are released. Solution uniqueness is enforced (App. C.2), which avoids penalizing valid alternative answers. Training-environment demonstration. The RL section, while preliminary, shows the benchmark’s reward signals are usable for training. The reward ablation (Obs-only vs. Oracle-only vs. full) is a sensible design. Weaknesses

  1. Several result patterns suggest possible evaluation or format artifacts rather than model failures

This is my main concern. Some patterns in the appendix tables are hard to explain as genuine reasoning failures.

Slitherlink transfer is 0.0 for every model. This includes models with 100% Oracle Acc. on Slitherlink anchors: Claude-Opus-4.6, Claude-Sonnet-4.6, Claude-Opus-4.7 and GPT-5.4@medium. The transfer pool includes 5×5, 7×7 and 10×10 boards (Table 15), the same sizes as the anchors (Table 14). A solver that is correct on a 5×5 anchor but wrong on every 5×5 transfer instance, for every model, points more toward an answer-canonicalization or format mismatch than toward “global loop reasoning” (L353–357). Examples would be the loop’s start vertex or direction, or the closing vertex. Masyu, also a closed-loop puzzle, reaches 94.3% transfer accuracy for GPT-5.4@high. Hashi shows the same pattern. Most models reach 100% Oracle on Hashi anchors but 0.0% transfer accuracy (Gemini-3.1-Pro, Gemini-3-Flash, both Claude-Opus-4.6 variants, Doubao, Qwen3.5-Plus), even though the transfer pool includes the anchor sizes 7×7 and 10×10. Only the GPT-5.4 variants succeed. A possible cause is bridge-key ordering in the answer dict, ((r1,c1),(r2,c2)) vs. reversed. Exec. Acc. exceeds Oracle Acc. on Skyscrapers for many models. Examples are Gemini-3.1-Pro (66.7 vs. 33.3), Kimi-K2.5 (44.4 vs. 11.1), Qwen3.6-Plus (44.4 vs. 22.2) and GPT-5.4@high (88.9 Exec with 33.3 Obs). A solver that succeeds on the model’s own (supposedly wrong) observation but fails on the ground-truth observation suggests the ground-truth observation differs in convention from what the rules or image imply. Obs. Acc. on Skyscrapers is 33.3% for nearly every model, which reinforces this. Region parsing is near 0% for almost all models (Table 3), and Star Battle (1-star) Obs. Acc. is 0.0 for almost every model despite non-zero Exec. Acc. It is unclear whether region-ID matching is invariant to label permutation. If not, the “region perception bottleneck” (L370–377) may partly be a labeling artifact. The Aquarium case study (App. D.10) shows genuinely different partitions in that instance, but that does not settle the general question. Claude’s token-level observation accuracy is surprisingly low. On simple token tasks it is 0.23 (Table 3), e.g., 16.7% on Binairo, while most other models score ≈1.0. Grok-4 is 0.06. That gap is large enough that it may reflect schema compliance (e.g., 'W' vs. 'w', None vs. ''), image preprocessing (resizing to 768 px), or an API-side image handling difference, rather than visual perception.

The paper’s headline diagnostic claims rest on these metrics: perception as the dominant bottleneck, region parsing near zero, and loop puzzles unsolved. The patterns above need to be explained or ruled out before the claims can be trusted.

  1. Limited statistical rigor Anchor counts per family range from 2 (Nonograms, Star Battle 2-star/shapeless) to 12 (Kakurasu). Per-family metrics therefore move in steps of 8–50 percentage points, and Cracked@30 or Master@30 can flip on a single instance. All results are single runs (temperature 0, but Kimi at 1.0), with no confidence intervals or seeds. Many comparisons in the text rest on differences of a few points, for example Table 9 (−3.2 / +2.1 pp) and Table 8. These are likely within noise. Verified@190 is micro-averaged over anchors, so families with more anchors weigh more. Rel. Eff. and Win Rate are macro-averaged over families. The weighting differs across metrics in the same table without justification.
  2. Code vs. text comparison (Fig. 1) is narrow and possibly not apples-to-apples

The comparison uses a single model at low reasoning effort (GPT-5.4@low, App. C.1). The 53.7% code figure on PuzzleCodeBench matches the diagnostic setting Exec. Acc. (Table 5), where the reference observation schema is provided. Yet Fig. 2 reports GPT-5.4@low end-to-end code accuracy as 37.9%, which is below the 43.2% text baseline.

The paper should state clearly which setting each number comes from. As presented, the claim that code is a better interface than text depends on that choice. A low-effort text baseline is also a weak comparison for long-chain puzzle solving.

  1. The end-to-end setting depends on an unspecified LLM reformatter

End-to-end outputs are passed through an LLM “format converter” (App. G) instructed to make “a strong best-effort parse.” The paper does not say which model performs this conversion, how often it intervenes, or whether it could repair or even supply answers. That makes the Fig. 2 headline numbers harder to interpret.

  1. RL claims are overstated relative to the evidence “Zero-shot generalization to 9 unseen families” rests on Cracked@30. That metric only requires one anchor in a family to be oracle-correct, and some families have only 2–3 anchors. Table 13 (unseen families only) shows 14.2% Oracle and 8.3% Exec, about 10 of 120 instances, with no pre-RL baseline on those same 19 families. The statement that the model “effectively induces robust, reusable algorithm design rather than merely overfitting” (L469–471) is not supported, especially since no transfer-level metric (Verified@190) is reported after RL. Data counts don’t add up. “1,000 additional training samples” vs. “10 anchor instances per family” over 11 families. Table 12 lists Qwen3.5-9B-Base, while the text uses the thinking-mode model. Contamination and baselines. It is unclear whether the 11 training families’ benchmark anchors and transfer instances were excluded from the generated training data, or whether only the generator was shared. The SFT baseline (App. E.1) lacks details on where its targets came from. RL is single-seed.
  2. Prompt design confounds what is measured

The prompts state that efficiency is “the ultimate metric,” prohibit “naive DFS/BFS,” and require “human-like deduction.” None of these constraints is verified. Strong efficiency language may push models toward fragile, over-engineered propagation code, contributing to the wrong answers seen on transfer. A neutral prompt (correctness first, no stylistic mandates) would help separate model capability from prompt pressure.

  1. “Extended reasoning” tiers are not compute-matched, and token accounting is unclear

Claude-Opus-4.6@max averages 3,283 output tokens vs. 3,120 at default, while GPT-5.4@high averages 24,228 (Table 10). Either Claude’s thinking tokens are not counted or “max effort” did not take effect.

GPT-5.4 “default” corresponds to reasoning_effort=none (App. C.3), while other providers’ defaults include some reasoning. Cross-vendor tier comparisons, e.g., “Increasing test-time reasoning helps narrow this gap,” should be framed with these caveats.

  1. Memorization and novelty

These are classic, widely implemented puzzle families; many are in Simon Tatham’s collection, and solvers for them are abundant online. Models may be recalling known solver code rather than synthesizing from rules. The limitations section acknowledges this, but it directly affects the “algorithmic reasoning” interpretation.

On novelty, the comparison in Table 1 omits the large line of work on LLMs generating executable programs or solver encodings for reasoning problems, such as PAL, Program-of-Thoughts and Logic-LM. Code-as-reasoning-interface is established. The contribution is its application to visual grid puzzles with transfer and efficiency evaluation, which should be positioned explicitly.

  1. Smaller inconsistencies Citations. TopoBench is cited as Telyatnikov et al. (2025) in Table 1 and the introduction, which is a topological deep learning framework. The appendix correctly cites Maniparambil et al. (2026). Anchor sizes. The text says anchors span 5×5 to 10×10 (L203), but Table 14 includes 4×4 (Futoshiki, Kakurasu, Renzoku, Skyscrapers) and 11×11 (Shingoki, Slant). The “up to 25×25” claim applies to only some families; Sudoku is 9×9 only and Skyscrapers ≤ 6×6. Same setting, different numbers. The GPT-5.4@low joint setting gives 54.2% Exec. in Table 8 vs. 53.7% in Table 5. Tied aggregates. Claude-Opus-4.6 and Claude-Opus-4.6@max have identical aggregate Obs. Acc. (23.7%) and identical Table 3 perception rows, despite different per-family values. Please confirm this is not a copy error. Untested reference solvers. Win Rate is relative to the authors’ reference solvers, whose quality is not independently validated. If they are weak, win rates are inflated.

The task formulation is valuable, and the Obs/Oracle/Transfer decomposition is a genuine contribution. However:

(i) Several prominent results show patterns more consistent with evaluation or format artifacts than with model failures: Slitherlink and Hashi transfer at 0% even at anchor sizes, and Exec > Oracle on Skyscrapers. The paper’s key diagnostic claims rest on exactly these numbers. (ii) The small, uneven per-family samples without uncertainty estimates, and the overstated RL generalization claims, weaken the empirical support.

If the authors can show the evaluator is sound on these families (or fix it and update the results), add uncertainty estimates, and temper the RL claims, I would view this as a solid benchmark paper suitable for acceptance.

Supporting arguments

A benchmark’s value lies in the trustworthiness of its measurements. The paper’s distinctive claims are of three kinds:

which capability is the bottleneck (perception, especially regions); which puzzle types defeat all models (Slitherlink, Yin-Yang, loops); how much solvers generalize (Verified@190).

Each of these depends on the per-family numbers flagged in Weakness 1. The anomalies are concentrated in specific families and cut across all models, which is the signature of a harness issue rather than a capability gap. Without resolution, practitioners could draw wrong conclusions, for example that loop reasoning is uniformly unsolved when the issue may be answer canonicalization.

The RL contribution is explicitly framed as preliminary. That is fine, but the stated conclusions should match that framing.

Questions for the authors Slitherlink and Hashi. Why is transfer accuracy 0.0 for models with 100% anchor Oracle Acc., including on transfer instances of the same size as the anchors? How are loop answers (start vertex, direction, closure) and Hashi bridge keys (endpoint order) canonicalized before comparison? Could you report transfer accuracy restricted to anchor-sized instances? Skyscrapers. How can Exec. Acc. exceed Oracle Acc. (e.g., Kimi 44.4 vs. 11.1)? Does the ground-truth observation use a convention (clue ordering, missing-clue encoding) that differs from what the image and rules imply? Observation matching. Is Obs. Acc. invariant to region-ID relabeling? Does it normalize equivalent encodings (None vs. '', case)? What fraction of observation errors are schema or formatting issues vs. genuine misreadings, especially for Claude and Grok on token tasks? End-to-end reformatter. Which model performs the format conversion? How often does it change or recover answers, and have you audited it for solving rather than reformatting? Fig. 1. Is the 53.7% code number the diagnostic setting? How does code compare to text under the true end-to-end setting (37.9% vs. 43.2% for GPT-5.4@low)? Can the comparison be extended to high reasoning effort and at least one more model? Uncertainty. Can you report confidence intervals (e.g., bootstrap over instances) and, for a few models, variance across multiple samples or seeds? RL. What is the pre-RL baseline on the 19 unseen families in Table 13? What are post-RL transfer metrics (Verified@190, Master@30)? How exactly were training instances separated from benchmark anchors and transfer pools? How do “1,000 samples” and “10 anchors per family” reconcile, and was the base model Qwen3.5-9B-Base or the instruct/thinking model? Token counts. Are Claude’s thinking tokens included in Table 10? Was the effort: max setting verified to take effect? Reference solvers. How strong are the reference solvers (e.g., compared against established puzzle solvers)? Are they implemented in pure Python like the generated solvers, so instruction-count ratios are comparable? Prompt sensitivity. How sensitive are the results to the efficiency- and “human-like deduction”-heavy prompt? A neutral-prompt ablation on a subset would help. Additional feedback (to help improve the paper, not part of my decision) Consider balancing the anchor set (equal anchors per family) or reporting both micro and macro averages consistently across all metrics. Report the mapping from each family to the perception types (numeric, token, region, symbol) used in Table 3. Add a short failure taxonomy based on manually inspected transfer failures (missing constraint vs. weak propagation vs. timeout). This would back the claims in L405–415 more directly than aggregate counts. Position the work against code-as-reasoning approaches (PAL, Program-of-Thoughts, Logic-LM) and visual program-synthesis benchmarks in the related-work discussion and Table 1. A contamination probe would strengthen the “algorithm synthesis” framing, e.g., rule variants or novel families not found online. Fix the TopoBench citation, the anchor-size statement, and the Table 5/Table 8 discrepancy. Fig. 2 lists a subset of models with slightly different names from Table 2 (“Claude-Opus-4.6” vs. “@max”). Aligning them would avoid confusion. Polish: “a complete, stateful, and automatically verifiable solving procedures” (L42).

Questions for authors

I'm not sure what figure 1 is showing - could you explain what experiments that is summarizing?

In a 2023 ICLR paper https://arxiv.org/abs/2207.14502 an approach for generating python puzzles and solutions was proposed, where the RL part used the model to generate more puzzles and solutions, and the restriction to problems and solutions being done in code helped the verifier ensure a puzzle solution match. How were the additional families of problems created for your RL work? Are you able to automate that at all like https://arxiv.org/abs/2207.14502 did?

4 comments
AuthorsOpenReview ↗

Response to Reviewer AoFo (Part 1/2)

We sincerely thank the reviewer for the insightful comments and the time and effort devoted to reviewing our paper. We would first like to address the primary weaknesses and questions, followed by responses to the key concerns raised in the LLM comments.

W1: Benchmark and RL Novelty

Thank you for recognizing the breadth of PuzzleCodeBench. As shown in Table 1, among the compared benchmarks, PuzzleCodeBench uniquely combines visual inputs with executable solver outputs, covering 30 grid-based puzzle families. Its novelty lies in bringing visual puzzle reasoning and visual-to-code generation together in a unified benchmark for reusable solver synthesis, rather than evaluating these capabilities separately or focusing solely on instance-level answers.

This formulation enables systematic evaluation of capabilities beyond conventional puzzle-solving benchmarks. By executing each generated solver on its anchor and 20 held-out instances of the same family, we evaluate not only visual perception and solver correctness, but also cross-instance generalization and computational efficiency. This assesses whether models can capture reusable puzzle rules and construct correct, computationally efficient algorithms that generalize across instances, rather than merely solving individual puzzles. Furthermore, disentangling visual perception and solver correctness reveals distinct model-specific bottlenecks and provides insights into their interplay.

We agree that our RL experiments do not introduce a novel RL algorithm. Rather, they serve as an exploratory demonstration that PuzzleCodeBench provides execution-grounded supervision for improving solver synthesis and visual puzzle solving. We hope this benchmark will encourage future work on more effective and novel RL methods for these challenging solver-synthesis tasks.

Q1: Clarification of Figure 1

We apologize for the lack of clarity in presenting Figure 1. Figure 1 provides empirical motivation for our visual-to-code formulation by comparing text-only and code-based puzzle solving using GPT-5.4@low on MM-HELIX and PuzzleCodeBench, showing consistent accuracy improvements with code-based solving. This is particularly relevant to grid-based puzzles, whose structured board states and complex constraints can be explicitly represented and handled through executable programs rather than direct answer generation. Furthermore, executable solvers allow us to evaluate cross-instance transfer, which we investigate separately through our anchor-transfer evaluation. We hope this clarifies the motivation and purpose of Figure 1.

Q2: Automated RL Data Generation

Thank you for highlighting this related work. Our RL model uses additional instances from 11 of the 30 predefined puzzle families, selected because the base model solved at least one anchor in each. Each family has a dedicated generator, reference solver, and renderer, enabling automatic generation of instances with varied boards, sizes, and difficulty levels, with solution verification and uniqueness checks, without manual annotation. Thus, our pipeline automates instance generation within predefined visual puzzle families, whereas the cited work uses language models to synthesize new programming puzzles and solutions. Extending our pipeline to automatically generate entirely new visual puzzle rules and families is an interesting direction that we will explore in future work.

AuthorsOpenReview ↗

Response to Reviewer AoFo (Part 2/2)

Responses to Key LLM comments

C1: Inconsistencies in the puzzle cards. Thank you for pointing these out. During our initial exploration of the 30 puzzle families, we used preliminary construction code with manually prepared examples. Some early rules and solutions were inadvertently carried over when preparing the repository and appendix, affecting Binairo+, Stitches, and Tents. We also inadvertently included outdated specifications for Skyscrapers (observation format) and LITS (Rule 3) in the appendix and repository. We have corrected these inconsistencies. These issues were confined to the preliminary examples and did not affect the benchmark instances or evaluation results.

C2: Equivalent representation matching and canonicalization. We use puzzle-specific test functions to check semantic equivalence rather than relying on raw representation equality. These functions handle region ID permutations, loop rotations and reversals, and unordered edge/stitch collections with normalized endpoint ordering.

C3: Strictness of the observation schema. Each puzzle's rules explicitly specify the observation format and allowed values. We strictly follow these specifications, treating alternative representations (e.g., None vs. '' or 'B' vs. 'b') as mismatches by design.

C4: Region parsing and observation accuracy. Our comparators normalize region IDs, and we found no false mismatches caused by different region labels. The low observation accuracy on Star Battle (1-star/2-star) mainly reflects genuine region parsing errors, while the much higher Oracle Accuracy shows that models can often solve these puzzles when given correct region layouts.

C5: Clarification of transfer evaluation and efficiency metrics. Transfer evaluation only requires the generated code to compile and execute successfully on the anchor; passing the anchor correctness check is not required. For efficiency evaluation, we measure performance on individual correctly solved transfer instances, rather than restricting evaluation to solvers that pass all 20 transfer cases. Incorrectly solved instances receive zero efficiency credit. We have clarified these distinctions in Sections 2.3 and 2.4.

C6: E2E Output Conversion. We use GPT-5.4@low only for answer extraction and format conversion, not for solving the puzzles or correcting the generated programs.

C7: Fairness of the code-versus-text comparison. Thank you for pointing this out. Fig. 1 uses the standardized diagnostic interface for code-based evaluation, whereas Fig. 2 reports end-to-end code generation without this interface. The two figures therefore use different evaluation settings and are not directly comparable.

C8: Minor writing and reporting inconsistencies. Thank you for pointing these out. We have corrected the TopoBench citation, the anchor grid-size range, the GPT-5.4@low result in Table 8 (53.7%), and the RL base model description (Qwen3.5-9B with Thinking mode). We have also fixed the grammatical errors and other minor inconsistencies.

C9: RL generalization beyond Cracked@30. Thank you for this suggestion. We additionally evaluate Verified and Master before and after RL, separating the 11 training families (70 anchors) and 19 held-out families (120 anchors), with each solver tested on 20 transfer instances.

On the 19 held-out families, RL improves Cracked from 0 to 9, Master from 0 to 2, and Verified from 0/120 to 5/120. On the 11 training families, Verified improves from 11/70 to 26/70, and Master from 6 to 8. These results provide a pre-RL baseline on the same held-out instances and demonstrate improved within-family and cross-family generalization, although generalization to unseen families remains challenging.

C10: Robustness to unequal numbers of anchor instances. Thank you for raising this concern. We conduct a balanced evaluation with exactly 6 anchors per family (180 in total), adding or subsampling instances as needed. Results on five representative models remain stable, with Execution accuracy changing by at most 2.6 percentage points and Master@30 by at most one family.

ModelObs. Acc.Exec. Acc.Oracle Acc.Verified@180Master@30
GPT-5.4@high82.2% (-2.0)70.0% (-2.6)81.1% (0.0)64.4% (+1.8)24 (0)
Gemini-3.1-Pro@high70.6% (-0.5)71.1% (-0.5)86.7% (+0.9)57.8% (-0.1)22 (-1)
Claude-Opus-4.620.6% (-3.1)17.8% (-0.6)76.7% (+0.4)42.2% (+1.7)19 (0)
GPT-5.470.6% (+0.6)39.4% (-0.1)47.2% (-0.7)30.6% (+1.1)17 (0)
Qwen3.6-Plus66.7% (+1.4)45.6% (+2.4)54.4% (+0.7)26.7% (+2.0)14 (0)
Reviewer_AoFoOpenReview ↗

Thanks for the responses

It's a good paper, I hope it gets accepted, I feel my scores are correct.

Thank you very much for your prompt response and encouraging feedback. We sincerely appreciate your support and constructive suggestions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.