- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper identifies silent errors in LLM-generated optimization models, where models achieve the correct objective value but violate the intended constraints or variable requirements. It proposes SILENTOR, a solver-backed verification framework that identifies requirement violations through executable probes and verifiable counterexamples. Experiments show that SILENTOR substantially improves requirement-localization precision and reduces false positives compared with LLM-based inspection.
Critical strengths
The paper identifies an important failure mode in LLM-generated optimization models. The concept of silent errors is insightful, and the motivating example and empirical analysis convincingly illustrate how correct objective values can mask incorrect formulations, exposing a meaningful limitation of existing evaluation practices.
Critical weaknesses
-
SILENTOR searches for candidate-feasible solutions that violate the specified requirements, making its verification inherently one-sided. This approach can detect errors that admit invalid solutions, but it cannot detect errors that only exclude valid solutions, including overly restrictive constraints and some incorrect constraints. Despite explicitly identifying these three categories of constraint errors, the paper neither establishes how its method handles the latter cases nor reports category-specific detection results.
-
SILENTOR relies on LLM-generated probes, while the solver verifies violations of the probes rather than the original requirements. Although judge-based screening and witness verification mitigate this issue, the false-positive rate of 30.6% with Gemma 3 12B highlights the framework's sensitivity to probe quality.
-
The multi-stage verification pipeline introduces additional LLM and solver calls, but the paper does not report its computational cost or discuss the trade-off between verification reliability and overhead. Its practical value therefore remains unclear.
-
The evaluation covers only 49 silent-error cases, including 25 naturally occurring cases and 24 controlled errors. The paper does not adequately explain why only 25 natural cases were selected for verification evaluation or establish that these cases are representative. Moreover, the evaluation relies on supplied, reviewed requirements rather than testing the complete verification pipeline in existing LLM-based OR modeling systems.
Questions for authors
-
How does SILENTOR detect overly restrictive constraints or incorrect constraints that only shrink the candidate feasible region? Could the authors report detection results separately for missing, incorrect, and overly restrictive constraints?
-
Could the authors quantify probe fidelity and analyze how incorrectly formulated probes contribute to false positives? How does the framework distinguish genuine modeling violations from errors in the probes themselves?
-
Could the authors report the number of LLM and solver calls, token consumption, and end-to-end runtime to clarify the trade-off between verification accuracy and computational overhead?
-
What criteria were used to select the 25 natural-error cases from the 236 silent-error cases identified in Section 3? Could the authors evaluate SILENTOR on outputs from existing LLM-based OR modeling systems to establish its practical effectiveness?
Thank you, and answers to your questions
Thank you for the careful review. We agree with your main points; below we answer each question using data already in the supplement.
Q1. Overrestrictive and shrinking incorrect constraints. We did not evaluate this direction: our evaluation set contains no overrestrictive cases, and we did not categorize or report results by error type. By design, SilentOR's witness search finds solutions the candidate admits but a requirement forbids, so it targets errors that enlarge the feasible region; an overrestrictive constraint, or an incorrect constraint that only excludes valid solutions, produces no witness. This is a one of the limitation.
Q2. Probe fidelity. The solver certifies violation of the probe, not of the requirement. Two steps try to separate the two: judges review each probe against the requirement before execution, and a witness verifier re-checks each witness against the requirement, rejecting but never creating a violation. Removing the witness check lowers precision from 96.2% to 90.9% (Appendix F). We did not measure probe fidelity directly, and the Gemma 3 12B results show strong dependence on the backbone.
Q3. Cost. We did not report computational cost, and we did not compare against a budget-matched alternative. In the paper, SilentOR verifies each candidate against its full requirement list. The method itself checks one requirement at a time (Algorithm 1), so it can also verify a single requirement or a chosen subset, with cost growing with the number of requirements checked rather than fixed per model.
Q4. Selection and generality. The 25 natural cases were chosen for coverage across problems and error types, since 67.8% of silent errors come from 16 problem–generator combinations that repeat the same mistake. Six of the 25 come from SIRL-Gurobi (Chen et al., 2025),; we did not evaluate other complete systems. We agree that 25 cases is small, that representativeness is not established, and that supplied requirements mean we evaluate verification rather than end-to-end use.
Thank you; these points identify the main limitations of the work.