- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes CROW (Constraint Reuse and Orchestration Workflow), which treats critical reasoning constraints in mathematics as knowledge objects that can be retrieved, constructed, verified, and reused. Given a problem, Qwen3-Embedding-0.6B retrieves the top-1 related problem from constraint memory. A Qwen3-4B orchestrator decides whether to read its constraint, construct a new one, directly invoke a natural-language/program-based solver, invoke the corresponding reviser, or submit an answer. Constraints can guide solving, verification, and revision. Newly constructed constraints used in successful trajectories and contributing to correct answers are added to memory. The orchestrator is trained with GRPO using final-answer correctness as a sparse terminal reward, while Qwen3-Coder-30B-A3B-Instruct remains frozen as the task model.
The main contributions are: (1) making critical mathematical conditions explicit as reasoning knowledge transferable across problems; (2) unifying decisions about when to retrieve, activate, construct, or disregard constraints, which solving modality to use, and when to revise into a sequential decision process learned through RL; (3) accumulating successful experience so that discovered constraints can be reused on later problems with similar structures; and (4) evaluation on HARDMath, HARDMath2, and the Precalculus/Calculus/Algebra subsets of DeepMath-103K, with component ablations and cost estimates.
Critical strengths
- Treating constraints as selectable, verifiable, and accumulable objects. This targets recurring structural mathematical errors more directly than simply increasing CoT, voting, or the number of agents. The same constraint is shared across generation, verification, and revision, and the motivation aligns with the failure cases.
- Relatively comprehensive component ablations showing complementary effects. Using constraints or memory alone leads to substantial degradation, while the complete system reaches 61.41, supporting the combination of reliable knowledge and selective orchestration rather than simply stacking modules.
Critical weaknesses
-
Insufficient controls to establish the method's contributions. In Table 2, adding memory improves HARDMath accuracy from 1.95% to 30.84%, but it remains unclear whether the gains come from knowledge retrieval or from constraint representation and dynamic orchestration. Direct retrieval and fixed-workflow controls using the same memory bank are needed.
-
Unclear memory construction and reliability. The source and number of initial seeds and the final memory size are not adequately specified. A correct final answer does not imply that the constraint or its generalized version is correct. The authors should explain the checks performed before memory insertion and assess the impact of incorrect constraints.
-
Limited evidence of generalization. The gains are mainly concentrated on HARDMath; HARDMath2 accuracy is 5.50%, below The Conductor's 10.00%. The authors should clarify overlap in problem types between training and testing and provide a split by problem type to assess whether the method relies on reusing highly similar problems.
-
Comparison budgets are not adequately matched. CROW uses 1,748 training problems, while search-based methods use only 100 optimization problems, and inference step limits also differ. Comparisons under matched budgets are needed, together with total cost and latency accounting for orchestration, retrieval, and program execution.
Questions for authors
- With the same memory bank, how do direct retrieval or a fixed workflow perform compared with CROW?
- What are the seed sources and memory size? How are constraints validated before insertion, and how are incorrect constraints handled?
- How much do training and test problem types overlap? Do the gains remain under a split by problem type?
- Do the conclusions hold under matched budgets? Please report full costs and end-to-end latency.
5 comments
Response to Reviewer nJog (2/4)
W2. Unclear memory construction and reliability. The source and number of initial seeds and the final memory size are not adequately specified. A correct final answer does not imply that the constraint or its generalized version is correct. The authors should explain the checks performed before memory insertion and assess the impact of incorrect constraints.
Q2. What are the seed sources and memory size? How are constraints validated before insertion, and how are incorrect constraints handled?
CROW starts with 9 seed entries constructed exclusively from training data and expands to 21 entries by the end of RL training, with 12 additional entries acquired automatically during training. For seed construction, we sample training problems and first attempt to solve each problem without constraint guidance. If direct solving succeeds, no memory is created. Otherwise, Qwen3-Coder samples 10 candidate constraints, and a problem-constraint pair is retained only when constraint-guided solving succeeds. We apply the same criterion during RL training: a newly proposed constraint is added to memory only when direct solving fails but using the constraint leads to a correct solution. We agree that this does not formally prove that a stored constraint or its generalized form is universally correct. Instead, the procedure filters for constraints that provide additional reasoning value on the training problem. At test time, the constraint associated with the retrieved problem is not automatically applied, the orchestrator can reject it, construct a new constraint, or solve directly when it is not suitable for the current problem. We will clarify the initial and final memory sizes, the insertion criterion, and how non-transferable constraints are handled in the revision. To answer W2 and Q2, CROW uses training-only seed memories, grows the memory during RL training, and inserts a constraint only when it adds reasoning value beyond direct solving, non-transferable constraints can be rejected at test time.
Response to Reviewer nJog (3/4)
W3. Limited evidence of generalization. The gains are mainly concentrated on HARDMath; HARDMath2 accuracy is 5.50%, below The Conductor's 10.00%. The authors should clarify overlap in problem types between training and testing and provide a split by problem type to assess whether the method relies on reusing highly similar problems.
Q3. How much do training and test problem types overlap? Do the gains remain under a split by problem type?
For HARDMath, we use the official predefined train/test split without modifying the instances. HARDMath contains six problem categories, and all six appear in both training and test sets, the split is therefore by instance rather than by problem category. We will clarify this scope in the revision. A separate problem-type split is not provided by HARDMath and would require introducing a new taxonomy outside the standard benchmark protocol. The original seed memory contained no experience from Precalculus, Calculus, or Algebra, so the orchestrator saw few successful examples of constraint use in these domains and increasingly favored direct solving, which also limited subsequent memory expansion. For each of these three domains, we add one constraint to the initial memory. We iterate through its training problems and first attempt direct solving. If direct solving succeeds, we move to the next problem, otherwise, Qwen3-Coder samples 10 candidate constraints, and the corresponding problem-constraint pair is retained once constraint-guided solving succeeds. All other settings remain unchanged, and memory continues to expand dynamically during RL training.
| Method | HARDMath | HARDMath2 | Precalculus | Calculus | Algebra | Overall |
|---|---|---|---|---|---|---|
| Conductor | 3.90 | 10.00 | 88.93 | 82.61 | 63.38 | 55.38 |
| CROW | 30.84 | 5.50 | 92.50 | 86.96 | 75.35 | 63.92 |
Against The Conductor, our strongest baseline under the same training split and data budget, CROW achieves higher accuracy on four of five benchmarks, with particularly clear gains on HARDMath and Algebra. This provides stronger evidence that CROW’s benefit generalizes beyond HARDMath, while HARDMath2 remains the one benchmark where The Conductor performs better. To answer W3 and Q3, we follow the official HARDMath split and the broader cross-domain results show that CROW improves beyond HARDMath.
Response to Reviewer nJog (1/4)
We thank the reviewer for the detailed feedback and for raising these concerns. We address each point below, combining related weaknesses and questions where they can be answered by the same evidence.
In CROW, Qwen3-Embedding-0.6B always retrieves the top-1 memory entry. Our memory-without-RL ablation uses the same latest memory after CROW training, frozen during evaluation, but replaces the GRPO-trained orchestrator with the base Qwen3-4B orchestrator. It achieves 43.54% overall accuracy, compared with 61.41% for full CROW. Without learned orchestration, the model is less reliable at deciding whether the retrieved memory is applicable, which can lead it either to misuse an unsuitable constraint or to ignore a useful one. We further isolate the constraint representation using the 401 test queries for which the orchestrator chooses to use the constraint associated with the retrieved problem. For each query, we only replace this constraint with the same retrieved problem and its successful solution, all other settings remain unchanged. The results are:
Constraints outperform demonstrations on all five benchmarks, with an overall improvement of 15.21%. Analysis of cases where demonstrations fail but constraints succeed shows that demonstrations may lead the solver to reuse instance-specific assumptions or revert to its habitual reasoning, while executable programs may run successfully despite encoding an invalid formulation. Constraints instead make the required reasoning condition explicit and can directly guide generation, verification, and revision.
Together, these controls disentangle the contributions of constraint representation and learned orchestration, showing that retrieval alone does not explain CROW's gains.
To answer W1 and Q1, the same-memory ablation and matched representation control show that CROW benefits from both learned orchestration and constraint representation, rather than retrieval alone.