acceptodds
Under review as a conference paper at ICLR 2027

From Demonstrations to Solver-Guided Exploration: Learning to Formulate Constraint Programs from Natural Language

Abstract

Translating natural-language descriptions into executable constraint programming (CP) models requires more than generating code that compiles. A model may return a solution while omitting a required constraint, using an incorrect domain, or optimizing the wrong objective. Human experts address such errors by inspecting solver feedback and revising the formulation, whereas existing LLM-based CP modeling systems often generate a model once or follow a fixed repair procedure. We introduce CPARL, a solver-grounded learning framework that combines CP-specific supervised fine-tuning with agentic reinforcement learning. The supervised stage first teaches direct modeling and solver-assisted revision through general demonstrations, followed by a second round targeting errors identified on a separate development set. Building on these capabilities, the reinforcement learning stage learns to choose between direct generation and solver-assisted revision, and to control model edits, solver operations, and termination. For tool-assisted trajectories, we propose Solver-Event Group Relative Policy Optimization (SE-GRPO), which groups decisions between successive solver evaluations and assigns credit using incremental solver evidence, anchored by semantic validation. We further introduce NL4CP, a new CP modeling benchmark built from formally specified problems, and evaluate CPARL on NL4CP and four public benchmarks. Using a Qwen3.5-27B backbone, CPARL achieves 35.0% micro and 44.6% macro pass@1 accuracy, exceeding all evaluated baselines fine-tuned for optimization modeling and directly prompted DeepSeek-V4-Pro and GLM-5.1 in both micro and macro averages. Ablation studies examine the contributions of the training stages, supervision mixture, and solver-event credit assignment. We analyze code executability and executable-but-invalid failures.

Primary area:
optimization
Keywords:
Optimization Modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LS9HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CPARL learns to translate natural-language problems into constraint programs through two rounds of supervised fine-tuning followed by solver-guided reinforcement learning. Its SE-GRPO objective assigns credit to blocks of edits and solver calls, while a shared outcome reward trains direct generation and route selection. The paper also introduces NL4CP, with solutions checked against formal XCSP3 specifications. Across five benchmarks, CPARL improves Qwen3.5-27B to 35.0% micro and 44.6% macro pass@1, with ablations examining supervision, routing, and credit assignment.

Critical strengths
  • NL4CP checks returned assignments against formal specifications, providing stronger validation than objective matching alone offers.
  • The staged training substantially improves the shared backbone across five benchmarks, supporting its practical utility.
  • The same-reward trajectory control helps separate block credit assignment from changes in the reward design.
Critical weaknesses
  • The small block-credit advantage remains uncertain because repeated evaluations vary decoding settings rather than training seeds.
  • Unequal feedback access and unclear interface compatibility limit the conclusions drawn from external baseline comparisons.
  • Unclear test-time validation and untested overconstraint risks limit confidence in the method's practical reliability.
Questions for authors
  1. Table 2 shows gains of only 0.8 micro and 0.9 macro percentage points over the same-reward trajectory control. Do these gains persist across independent training runs under a common checkpoint-selection rule?

  2. Section 5.1 compares CPARL with baselines lacking intermediate solver feedback. How much advantage remains over a fixed repair loop using the same SFT model, feedback, and generation budget?

  3. Section 4.2 includes validation feedback during interactive generation. Is validate callable at inference, and does its feedback depend on benchmark references?

  4. Successful output validation need not exclude overconstrained models rewarded by Equation (2). Have accepted, heavily pruned programs been audited for unintended restrictions, or evaluated with propagation credit removed?

  5. For the near-zero baseline scores in Table 1, how often do failures reflect output-format incompatibility, program execution, or incorrect modeling?

  6. The captions for Tables 1 and 2 appear below the tables; move them above to follow the required table format. Enlarge Table 2's text so that its means and standard deviations are legible.

7 comments
AuthorsOpenReview ↗

Response to Reviewer LS9H — Part 1/4

We thank the reviewer for recognizing NL4CP's solution checks against formal specifications, the practical gains from staged training across five benchmarks, and the value of the same-reward trajectory control for isolating credit assignment. We also appreciate the concerns about independent training stability, fair baseline comparisons, validation access, and the reliability of propagation rewards. We address each point with additional experiments and analysis below.

Response to W1/Q1. Independent training stability and common checkpoint selection

(Q1) “Table 2 shows gains of only 0.8 micro and 0.9 macro percentage points over the same-reward trajectory control. Do these gains persist across independent training runs under a common checkpoint-selection rule?”

We conducted 15 independent RL runs: SE-GRPO, outcome-only GRPO, and the same-reward trajectory-level-credit control (E3), each with training seeds 17, 137, 2027, 3407, and 7919. All runs start from the same two-round SFT checkpoint. For each method and training seed, we select its own best checkpoint using the same Dev-1000 pass@1 criterion:

Training seedSE-GRPOOutcome-only GRPOE3
17150225180
137165180150
2027150150135
3407180165180
7919150150150

Each selected checkpoint is evaluated with five evaluation seeds on all 900 test instances. The table reports mean ± sample SD of Micro accuracy in percent. SD describes evaluation variability at each fixed checkpoint; separate rows correspond to independent training runs.

Training seedSE-GRPOOutcome-only GRPOE3
1736.51 ± 1.2334.22 ± 0.3436.18 ± 0.35
13735.22 ± 0.7334.64 ± 0.8733.87 ± 0.52
202735.96 ± 0.6534.60 ± 0.7435.56 ± 0.42
340735.82 ± 0.9335.20 ± 1.0534.18 ± 1.28
791934.60 ± 0.2633.82 ± 0.2834.56 ± 0.26

We acknowledge that the original gains over E3 of 0.8 and 0.9 percentage points are modest. However, with independent training runs and each method selecting its own checkpoint under the same development criterion, SE-GRPO achieves higher mean Micro accuracy than E3 for all five matched training seeds. Averaging the five training-seed means gives 35.62% for SE-GRPO and 34.87% for E3, a mean improvement of 0.75 percentage points. These results support stable gains from SE-GRPO, rather than an advantage arising solely from the original common-checkpoint comparison. See Appendix K.1 for details.

AuthorsOpenReview ↗

Response to Reviewer LS9H — Part 2/4

Response to W2.1/Q2. A fixed repair workflow with the same SFT model

(Q2) “Section 5.1 compares CPARL with baselines lacking intermediate solver feedback. How much advantage remains over a fixed repair loop using the same SFT model, feedback, and generation budget?”

To further demonstrate the importance of teaching the model to decide how to use tools, we added an evaluation of the same two-round SFT checkpoint with the predefined workflow construct → compile → repair if needed → propagate → repair if needed → solve → submit. This allows the model to revise its program based on solver feedback, while the operation order is fixed in advance. Key pass@1 results on all 900 test instances are below, reported as mean ± sample SD:

MethodMicro (%)Macro (%)
Two-round SFT + fixed tool workflow31.44 ± 0.1639.59 ± 0.52
Two-round SFT + autonomous interaction32.5 ± 0.441.1 ± 0.3
Full CPARL (SFT + RL)35.0 ± 0.944.6 ± 1.5

The same SFT checkpoint with autonomous interaction exceeds the fixed workflow by 1.06 Micro and 1.51 Macro percentage points; full CPARL exceeds it by 3.56 and 5.01 points, respectively. This supports learning tool timing and interaction control.

We also evaluated strong general-purpose models with solver feedback. For example, Claude-Sonnet-4.6 achieves 58.8 ± 2.2% Micro and 62.2 ± 2.8% Macro on all 900 instances across repeated full-test-set evaluations with different evaluation seeds. These results further illustrate the value of tool feedback for general-purpose models' CP modeling. See Appendix K.2 for the settings and results.

AuthorsOpenReview ↗

Response to Reviewer LS9H — Part 3/4

Response to W2.2/Q5. Baseline failure attribution and execution-error subtypes

(Q5) “For the near-zero baseline scores in Table 1, how often do failures reflect output-format incompatibility, program execution, or incorrect modeling?”

We analyzed execution logs and checker reports for the five fine-tuned baselines across all 900 test instances, distinguishing program execution errors, output-format errors, timeouts, validation failures or objective mismatches, and other errors. Under CPMpy + OR-Tools, program execution errors are the main failure cause for all five models:

ModelProgram execution errors as a share of failed cases
ORLM-8B93.5%
OptMATH-7B75.9%
OptMATH-32B89.7%
SIRL-7B82.1%
SIRL-32B85.8%

These percentages use failed cases as the denominator. Output-format errors account for a smaller share: for example, 5.5%, 7.6%, and 7.5% for OptMATH-7B, OptMATH-32B, and SIRL-7B, respectively. We further categorize execution errors into syntax/name errors, API errors, type/expression errors, data-binding errors, and other runtime errors.

A further breakdown by model and dataset reveals different sources of execution errors. Key results are below; each percentage uses program execution errors for the corresponding model and dataset as its denominator:

ModelMain execution-error patternsKey results
ORLM-8BSyntax/name errors dominate on several datasets; other runtime errors are prominent on CP-BenchSyntax/name errors account for 93.6% on TEXT2ZINC and 68.8% on CPEVAL; other runtime errors account for 69.7% on CP-Bench
OptMATH-7BAPI errors dominate on most datasets, alongside syntax and data-binding problemsAPI errors account for 78.0% on CPEVAL; syntax/name errors account for 44.1% on IndusCP; data-binding errors account for 16.3% on NL4CP
OptMATH-32BAPI errors are the largest category on every datasetAPI errors account for 96.8% on CPEVAL and 73.1% on NL4CP
SIRL-7BAPI errors are the largest category on every dataset, with substantial type/expression errors on some datasetsAPI errors account for 80.6% on CPEVAL and 70.0% on TEXT2ZINC; type/expression errors account for 28.2% on IndusCP
SIRL-32BBoth API and type/expression errors are prominentAPI and type/expression errors account for 44.2% and 47.1% on NL4CP, respectively; type/expression errors account for 34.0% on IndusCP

This breakdown shows that failures concentrate in code syntax, solver API usage, and type/expression construction, with clear differences across models and datasets. The low scores therefore cannot be explained solely by output-format incompatibility: reliably generating executable CP solving code remains a major bottleneck for these baselines under the current evaluation settings. See Appendix K.4 for the complete error analysis.

Reviewer_cKzxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CPARL, a framework for translating natural-language descriptions into executable constraint programming models. It combines two rounds of supervised fine-tuning with agentic reinforcement learning. The supervised stage teaches direct model generation and solver-assisted revision, followed by targeted training on adaptations of development-set failures. During reinforcement learning, the model learns whether to use tools, how to modify its program, which solver operations to request, and when to terminate.

Critical strengths
  • The paper correctly identifies that compilation and successful solver execution do not establish that a generated program faithfully represents the intended problem. Grouping edits and tool decisions according to solver evaluation events is a reasonable way to connect modeling actions with observable feedback. Binding propagation credit to subsequent validation of the same model version, while penalizing redundant interactions, is also a thoughtful design. These choices provide a concrete connection between the learning objective and the structure of constraint modeling.
  • The framework substantially improves the underlying backbone across all five benchmarks. The experiments include meaningful controls for answer-only, reasoning-only, tool-only, mixed supervision, staged versus pooled training, and trajectory-level credit under the same event rewards. These comparisons help distinguish several sources of improvement. NL4CP also contributes useful evaluation tasks grounded in formal source instances and independently checked assignments and objectives, providing stronger evidence than executability alone.
Critical weaknesses
  • The closest controlled comparison is the trajectory-level-credit variant, which retains the same event rewards and mixed-route losses. Full CPARL improves over this control by only 0.8 percentage points in micro accuracy and 0.9 points in macro accuracy (Table 2). Appendix H.6 further reports that SE-GRPO outperforms this control at eight checkpoints, ties at one, and underperforms at six. At the final checkpoint, the trajectory-level control solves 323/900 instances compared with 315/900 for SE-GRPO in the single-evaluation sweep.

  • The repeated evaluations vary decoding at a fixed checkpoint rather than independently repeating training. Moreover, all controls are compared at the checkpoint selected for CPARL, rather than selecting each method through an equivalent development-set procedure. Consequently, the evidence establishes a useful overall pipeline more convincingly than a stable advantage attributable specifically to the proposed credit-assignment mechanism. This distinction matters because SE-GRPO is a central methodological contribution.

Questions for authors
  • Please provide clearer demonstration on the motivation, especially why RL is necessary to improve before we review the harness based methods to improve the instruction that does not work here.
  • Does SE-GRPO retain its advantage over the same-reward trajectory-level-credit control across independent training seeds and when each method’s checkpoint is selected using the same development-set criterion? Given the changing ordering across checkpoints, this comparison would materially affect my assessment of the central methodological contribution.
3 comments
AuthorsOpenReview ↗

Response to Reviewer cKzx — Part 1/2

We thank the reviewer for recognizing the design of solver-event credit assignment, model-version-bound propagation rewards, and redundant-interaction penalties, as well as the gains across five benchmarks, the internal controls, and the evaluation value of NL4CP. We also appreciate the concerns about the motivation for RL, independent training stability, and fair checkpoint selection. We address each point with additional experiments and analysis below.

Response to Q1. Motivation for RL and its distinction from a fixed tool workflow

(Q1) “Please provide clearer demonstration on the motivation, especially why RL is necessary to improve…”

Tool feedback and policy training. Solver feedback can help repair errors, but providing tools and revision instructions does not itself determine when to call a tool, how to revise constraints based on its feedback, or when to stop. In particular, successful compilation and execution can still accompany incorrect constraints, domains, or objectives. Our motivation for RL is to improve these decisions through feedback on the model's own trajectories after two-round SFT has taught modeling and revision. SE-GRPO assigns credit to the edits and tool decisions between successive solver evaluations, connecting subsequent feedback to the relevant decisions.

To further demonstrate the importance of teaching the model to decide how to use tools, we added an evaluation of the same two-round SFT checkpoint with the predefined workflow construct → compile → repair if needed → propagate → repair if needed → solve → submit. This allows the model to revise its program based on solver feedback, while the operation order is fixed in advance. Key pass@1 results on all 900 test instances are:

MethodMicro (%)Macro (%)
Two-round SFT + fixed tool workflow31.44 ± 0.1639.59 ± 0.52
Two-round SFT + autonomous interaction32.5 ± 0.441.1 ± 0.3
Full CPARL (SFT + RL)35.0 ± 0.944.6 ± 1.5

The same SFT checkpoint performs better with autonomous interaction than with the fixed workflow, supporting state-dependent choices of tool timing and revision steps. With the same backbone and solver-tool interfaces, RL further improves over autonomous-interaction SFT by 2.5 Micro and 3.5 Macro percentage points. These results support learning the interaction policy on top of existing solver feedback and supervision. See Appendix K.2 for details.

AuthorsOpenReview ↗

Response to Reviewer cKzx — Part 2/2

Response to W1/W2/Q2. Independent training stability and fair checkpoint selection

(W2) “The repeated evaluations vary decoding at a fixed checkpoint rather than independently repeating training.”

(Q2) “Does SE-GRPO retain its advantage over the same-reward trajectory-level-credit control across independent training seeds and when each method’s checkpoint is selected using the same development-set criterion? Given the changing ordering across checkpoints, this comparison would materially affect my assessment of the central methodological contribution.”

Independent runs and common selection. We agree that repeated decoding evaluations at a fixed checkpoint do not replace independent training runs, and that evaluating every method at CPARL's selected checkpoint does not fully establish performance under fair selection. We therefore conducted 15 independent RL runs: SE-GRPO, outcome-only GRPO, and the same-reward trajectory-level-credit control (E3), each with training seeds 17, 137, 2027, 3407, and 7919. All runs start from the same two-round SFT checkpoint.

For each method and training seed, we select its own best checkpoint using the same Dev-1000 pass@1 criterion. The selected checkpoints for all 15 runs are:

Training seedSE-GRPOOutcome-only GRPOE3
17150225180
137165180150
2027150150135
3407180165180
7919150150150

Multi-seed evaluation at the best checkpoints. Each selected checkpoint is evaluated with five evaluation seeds on all 900 test instances. The table reports mean ± sample SD of Micro accuracy in percent. The SD describes variation across five evaluations at each fixed checkpoint, while the five training-seed rows report independent training results.

Training seedSE-GRPOOutcome-only GRPOE3
1736.51 ± 1.2334.22 ± 0.3436.18 ± 0.35
13735.22 ± 0.7334.64 ± 0.8733.87 ± 0.52
202735.96 ± 0.6534.60 ± 0.7435.56 ± 0.42
340735.82 ± 0.9335.20 ± 1.0534.18 ± 1.28
791934.60 ± 0.2633.82 ± 0.2834.56 ± 0.26

Gain magnitude and checkpoint rankings. We acknowledge that the original gains over E3 of 0.8 Micro and 0.9 Macro percentage points are modest. The original checkpoint sweep's 8 wins, 1 tie, and 6 losses also raise concerns about the stability of SE-GRPO's performance. However, with independent training runs and each method selecting its own checkpoint under the same development criterion, SE-GRPO achieves higher mean Micro accuracy than E3 for all five matched training seeds. Averaging the five training-seed means gives 35.62% for SE-GRPO and 34.87% for E3, a mean improvement of 0.75 percentage points. These results support stable gains from SE-GRPO, rather than an advantage arising solely from the original common-checkpoint comparison. See Appendix K.1 for details.

Dear Reviewer cKzx,

Thank you for your careful review and valuable feedback. Two days ago, we posted our replies to your comments and reported the results of additional experiments. As the rebuttal period draws to a close, we hope you’ll have a chance to read our replies and consider the additional experimental results in Appendix K. If you have any further questions, please let us know. We will do our best to address them before the discussion period ends.

Thank you again for your time and help!

Reviewer_tbSXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces CPARL, a framework for translating natural-language problem descriptions into executable constraint programming models. CPARL combines two rounds of supervised fine-tuning with agentic reinforcement learning. The first SFT round teaches direct model generation and solver-assisted revision, while the second targets errors identified on a development set. During RL, the model learns whether to generate a program directly or interact with a solver. The proposed Solver-Event GRPO (SE-GRPO) assigns credit to blocks of edits and tool decisions between successive solver evaluations, using changes in validation, executability, and propagation evidence. The paper also introduces NL4CP, a benchmark derived from formally specified XCSP3 problems. Across five benchmarks, CPARL achieves 35.0% micro and 44.6% macro pass@1 accuracy, compared with 32.5%/41.1% for two-round SFT and 34.0%/42.5% for outcome-only GRPO.

Critical strengths

Important problem and careful distinction between executability and correctness. The paper addresses a practically relevant challenge: a generated CP model may execute successfully while omitting constraints, using incorrect domains, or optimizing the wrong objective. It explicitly distinguishes full formulation equivalence from acceptance by an output checker, avoiding the assumption that solver success establishes semantic correctness.

Valuable training resources and independently grounded benchmark construction. CPARL-SFT-8K provides verified demonstrations of both direct modeling and solver-assisted revision, including targeted examples addressing observed failures. NL4CP derives natural-language tasks from formal XCSP3 specifications and independently checks returned assignments and objective values, providing a useful evaluation resource.

Informative internal ablations. The experiments compare supervision formats, demonstration mixtures, staged versus pooled SFT, outcome-only GRPO, forced tool use, and trajectory-level credit under the same event rewards. The same-reward trajectory-level control is particularly useful for separating the contribution of credit assignment from that of reward design.

Transparent discussion of implementation details and empirical limitations. The appendices document singleton-group handling, late-depth fallback advantages, checkpoint trajectories, and the distinction between decoding variability and independent training variability. They also acknowledge cases where trajectory-level credit outperforms SE-GRPO and clarify that the unified CPMpy evaluation does not reproduce the original benchmark systems.

Critical weaknesses

The incremental benefit of SE-GRPO is modest and not established across independent training runs. Relative to the same-reward trajectory-level control, full CPARL improves micro and macro accuracy by only 0.8 and 0.9 percentage points, respectively. The reported repeated evaluations vary decoding at fixed checkpoints rather than training seeds. Moreover, checkpoint rankings are inconsistent: SE-GRPO exceeds the trajectory-level control at eight checkpoints, ties at one, and trails at six; the trajectory-level control performs better at the final checkpoint. All methods are compared at the checkpoint selected for CPARL rather than being independently selected using an identical development criterion.

External comparisons do not isolate learned solver interaction from unequal tool access and language adaptation. General-purpose baselines receive no intermediate solver feedback, whereas CPARL can perform multi-turn solver-assisted revision. Consequently, the comparison combines the effects of training, tool access, additional interaction, and learned control. Optimization-specialized baselines are also required to generate CPMpy, despite potential differences from their native training interfaces. Missing comparisons against strong models or SFT policies using the same tools and a fixed repair procedure weaken the evidence that learned interaction is preferable to predefined workflows.

The inference-time validation interface and the semantic reliability of reward shaping require clarification. The framework allows validation feedback, including counterexamples, to guide revisions, but the relationship between this tool and the hidden benchmark checker is insufficiently clear. If inference-time validation queries reference specifications, the evaluation measures oracle-assisted modeling and should provide equivalent access to baselines. Furthermore, gating propagation rewards on output validation does not establish formulation equivalence: an overconstrained model may return a valid solution while excluding other legitimate assignments, potentially receiving misleading propagation credit.

Data separation and reward configuration are not sufficiently auditable. Training and evaluation draw from overlapping upstream resources, including CSPLib, NLP4LP, and XCSP. Although the authors describe textual and formulation-level screening, the paper does not provide a sufficiently detailed source-family overlap audit or a strict family-disjoint evaluation. The development set also supplies parent problems for later training adaptations while supporting several selection decisions. In addition, the manuscript does not clearly enumerate the final event-reward coefficients, propagation threshold, complexity weights, and operation costs, limiting reproducibility.

Questions for authors

Regarding Weakness 1: Could you report independent training-seed results for SE-GRPO, outcome-only GRPO, and the same-reward trajectory-level control, with each method selecting its checkpoint using the same development criterion? How should the claimed benefit of event-level credit be interpreted given the changing checkpoint rankings and the stronger final-checkpoint result of the trajectory-level control?

Regarding Weakness 2: Could you evaluate strong general-purpose models and the two-round SFT policy with the same solver/validator interface, feedback, and interaction budgets as CPARL, including a fixed compile–solve–repair baseline? Could you also assess whether the low scores of optimization-specialized models primarily reflect CPMpy interface transfer rather than modeling capability?

Regarding Weakness 3: What information does the inference-time validate operation access, and how does it relate to the hidden scoring checker? Could you provide an evaluation without reference-based validation and stress tests involving overconstrained formulations, omitted constraints, hard-coded feasible assignments, and artificial domain narrowing?

Regarding Weakness 4: Could you release source and problem-family identifiers, a train–development–test overlap audit, and results on strictly held-out problem families? Please also specify the final reward coefficients, propagation threshold, complexity weights, operation costs, and their selection ranges.

6 comments
AuthorsOpenReview ↗

Response to Reviewer tbSX — Part 1/4

We thank the reviewer for recognizing the practical relevance of the problem, the verified training resources, the internal controls, and the transparent discussion of limitations. We also thank the reviewer for identifying several important shortcomings in the original submission. We added experiments and analyses addressing each point, as detailed below.

Response to W1/Q1. Independent training stability and fair checkpoint selection

(W1) “The incremental benefit of SE-GRPO is modest and not established across independent training runs.”

(W1) “All methods are compared at the checkpoint selected for CPARL rather than being independently selected using an identical development criterion.”

Independent runs and common selection. We conducted 15 independent RL runs: SE-GRPO, outcome-only GRPO, and the same-reward trajectory-level control (E3), each with training seeds 17, 137, 2027, 3407, and 7919. All runs start from the same two-round SFT checkpoint. For each method and training seed, we select its own best checkpoint using the same Dev-1000 pass@1 criterion. This addresses both independent training variability and the comparison bias from using CPARL's selected checkpoint for every method. The selected checkpoints for all 15 runs are:

Training seedSE-GRPOOutcome-only GRPOE3
17150225180
137165180150
2027150150135
3407180165180
7919150150150

Multi-seed evaluation at the best checkpoints. Each selected checkpoint is evaluated with five evaluation seeds on all 900 test instances. The table reports mean ± sample SD of Micro accuracy in percent; the SD describes variation across these five evaluations at each fixed checkpoint.

Training seedSE-GRPOOutcome-only GRPOE3
1736.51 ± 1.2334.22 ± 0.3436.18 ± 0.35
13735.22 ± 0.7334.64 ± 0.8733.87 ± 0.52
202735.96 ± 0.6534.60 ± 0.7435.56 ± 0.42
340735.82 ± 0.9335.20 ± 1.0534.18 ± 1.28
791934.60 ± 0.2633.82 ± 0.2834.56 ± 0.26

Interpretation of checkpoint rankings. We agree that the original 8 wins, 1 tie, and 6 losses, including E3's stronger final checkpoint, preclude claiming uniformly better learning dynamics. However, with multiple training seeds and a common checkpoint-selection rule, SE-GRPO achieves the highest mean Micro accuracy at its own best checkpoint for all five matched training seeds. These results support consistent gains from SE-GRPO across independent training runs. See Appendix K.1 for details.

AuthorsOpenReview ↗

Response to Reviewer tbSX — Part 2/4

Response to W2.1/Q2. Strong general-purpose models and two-round SFT with fixed solver feedback

(W2) “General-purpose baselines receive no intermediate solver feedback, whereas CPARL can perform multi-turn solver-assisted revision.”

We agree that the original comparison combines the effects of training, tool access, and interaction. We therefore added tool-assisted evaluations of strong general-purpose models using construct → compile → repair if needed → propagate → repair if needed → solve → submit. This follows an intuitive expert checking sequence while allowing revisions based on solver feedback. Key pass@1 results on all 900 test instances are:

Tool-assisted modelMicro (%)Macro (%)
DeepSeek-V4-Pro47.7 ± 3.551.2 ± 5.0
GLM-5.144.2 ± 2.251.7 ± 2.0
GPT-5.455.5 ± 1.859.4 ± 2.5
Gemini-3.1-Pro54.3 ± 5.460.0 ± 6.4
Claude-Sonnet-4.658.8 ± 2.262.2 ± 2.8

Each result comes from multiple evaluations with different random seeds, each covering all 900 instances, and reports the mean ± sample SD. These results supplement the CP modeling performance of strong general-purpose models with solver feedback. This result shows that our tool feedback can not only significantly improve the modeling ability of fine-tuned models but also further improve the CP modeling ability of general-purpose large language models.

(W2) “Missing comparisons against strong models or SFT policies using the same tools and a fixed repair procedure weaken the evidence that learned interaction is preferable to predefined workflows.”

We evaluated the same two-round SFT checkpoint with the predefined workflow construct → compile → repair if needed → propagate → repair if needed → solve → submit. Across five evaluation seeds on all 900 instances, it achieves 31.44 ± 0.16% Micro and 39.59 ± 0.52% Macro, below two-round SFT with unrestricted interaction (32.5%/41.1%) and full CPARL (35.0%/44.6%).

The same SFT checkpoint performs better with autonomous interaction than with this fixed workflow, supporting the value of learning tool-use timing and interaction control. The gap between the fixed workflow and full CPARL also includes the effect of RL training.

Response to W2.2/Q2. Native interfaces for optimization-specialized models

(Q2) “Could you also assess whether the low scores of optimization-specialized models primarily reflect CPMpy interface transfer rather than modeling capability?”

We added native-interface evaluations: ORLM uses COPT; OptMATH and SIRL use Gurobi. The compact results below report pass@1 mean ± sample SD over five evaluations on the same 900 instances.

ModelNative Micro (%)Native Macro (%)
ORLM-8B6.64 ± 0.145.90 ± 0.60
OptMATH-7B3.49 ± 0.914.82 ± 1.26
OptMATH-32B4.78 ± 0.336.78 ± 0.53
SIRL-7B7.44 ± 0.226.44 ± 0.42
SIRL-32B14.87 ± 0.4019.15 ± 0.57

The results show that all five fine-tuned baselines improve their CP modeling performance when using their native interfaces, but still perform far below our models fine-tuned and trained with reinforcement learning specifically for CP modeling.

We also analyzed the causes of errors made by these five fine-tuned baselines on all 900 test instances. Program execution errors remain the main cause of failure with both CPMpy + OR-Tools and the models' native interfaces, although native interfaces reduce their share to some extent. This indicates that existing fine-tuned models are unfamiliar with CP modeling and cannot consistently generate executable CP-solving code. Details are provided in Appendix K.4.

AuthorsOpenReview ↗

Response to Reviewer tbSX — Part 3/4

Response to W3.1/Q3. Validation access and evaluation without reference-based feedback

(Q3) “What information does the inference-time validate operation access, and how does it relate to the hidden scoring checker?”

(Q3) “Could you provide an evaluation without reference-based validation…”

We clarify that inference-time validate uses only the small-scale example inputs and reference answers publicly provided in the problem; this operation is unavailable when such examples are absent. It must be made clear that it cannot query hidden formal models, hidden XCSP3 specifications, private optimal values, or the hidden scoring checker. The checkers and reference answers used to evaluate model performance are not provided to the model either. We have made this definition explicit in the paper (Appendix H.2).

We designed this tool because some large-scale CP modeling problems provide small test examples. In fact, none of the problems in the five CP datasets used in our evaluation provide such examples, so we did not invoke this tool during the actual solving and evaluation process.

Response to W3.2/Q3. Semantic limits of propagation rewards and stress tests for four semantic failure modes

(W3) “Furthermore, gating propagation rewards on output validation does not establish formulation equivalence: an overconstrained model may return a valid solution while excluding other legitimate assignments, potentially receiving misleading propagation credit.”

(Q3) “…stress tests involving overconstrained formulations, omitted constraints, hard-coded feasible assignments, and artificial domain narrowing?”

We agree that overconstrained models or models with artificially narrowed domains may still pass the gate. We therefore position propagation as a heuristic training signal, rather than a certificate of semantic correctness. Capping the pruning reward can also limit its magnitude, helping to mitigate this limitation.

To test the overconstraint, omitted constraints, hard-coded feasible assignments, and artificial domain narrowing you raised, we constructed four targeted diagnostic sets of 50 instances each, covering ten mathematical structures. The key checks and CPARL results are as follows:

Failure modeDiagnostic evidenceCPARL error rateTargeted errors as a proportion of all errors
OverconstraintRejects an independently verified legal assignment16%12.5%
Omitted constraintsAccepts an independently verified illegal assignment24%25%
Hard-coded feasible assignmentsRejects alternative legal assignments or fails under parameter changes with unchanged code; inspect code to confirm hard-coding18%0%
Artificial domain narrowingExcludes a genuinely legal value or assignment; inspect domains/code to establish the cause10%0%

Manual inspection of the stress-test results shows that CPARL does not exhibit hard-coded feasible assignments or artificial domain narrowing, and there is only one case of overconstraint. Although three omitted-constraint errors occur, these are common optimization-modeling errors and do not indicate a major defect in the model. We therefore consider it unlikely that CPARL's reward design induces the model to make errors.

Details of the stress-test set construction and manual inspection are provided in the paper (Appendix K.5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.