- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces OR-INSPECT-BENCH, a trajectory-aware benchmark and evaluation harness. It tests LLM agents on multi-turn Operations Research (OR) repair tasks with rich verifier feedback. The tasks cover linear and mixed-integer linear programming formulations. Most prior work studies one-shot translation from text to formulation. The authors study iterative debugging instead. To support this, they propose two interface-level components:
Repair-Sufficient State (RSS). A middleware parser that compiles raw solver logs, domain checks, and constraint conflicts into a structured five-channel state.
Counterfactual Verifier Loop (CVL). An inference-time search controller. It uses UCT and verifier transitions to choose diagnostic probes and candidate repairs.
The authors run experiments across 23 model snapshots, 360 hardened benchmark cases, and five external public benchmark suites. Their component analysis shows how typed verifier feedback and output handling affect agentic recovery.
Critical strengths
1. Shift from one-shot to iterative debugging. Prior benchmarks such as NL4Opt and MAMO test single-pass translation. This paper tests multi-turn error correction with exact solver feedback. That is a real bottleneck in practical optimization modeling.
2. Rigorous and transparent ablations. The authors deserve credit for the matched three-arm studies and factorials in Section 5.2. They isolate the performance gain from output normalization and the gain from search control, and report each separately.
3. Comprehensive evaluation design. OR-INSPECT-BENCH combines static diagnosis (TRACEJUDGE) with live multi-turn correction (REPAIRRUN). Both run under controlled perturbation types and verification gates.
Critical weaknesses
1. The paper’s own controlled experiments show that the headline search controller (CVL-r2) adds only +3.1% over a baseline with a reference-free output normalizer. That gain is not statistically significant; the 95% confidence interval includes zero. The normalizer alone adds +12.2%. In vast majority of cases, the CVL-r2 loop runs a single LLM decision. The search architecture is rarely exercised.
2. The benchmark instances are built from clean textbook problems with five fixed, pre-declared error types injected (e.g. objective flip, missing capacity). The agent’s operator schema matches those same five perturbations. This reduces open-ended mathematical modeling to multiple-choice classification plus slot filling. Once it is constrained into multiple-choice, the problem is much less challenging.
3. The paper leans on proprietary acronyms (RSS, CVL, OR-INSPECT-BENCH, TRACEJUDGE). Some of them obscure standard software engineering and compiler-harness concepts.
Questions for authors
1. Given that adding the reference-free output normalizer to RSS yields a major performance leap (+12.2%) while adding the full CVL-r2 controller yields a statistically insignificant +3.1% gain (and only makes a single decision in 97.5% of cases), how do the authors justify the architectural and computational overhead of the CVL search tree?
2. How do the authors expect agents evaluated on OR-INSPECT-BENCH's closed-world catalog of 5 synthetic perturbations to generalize to open-ended structural formulation errors in real-world industrial optimization tasks?
3. Since this paper covers 2 out of several OR methods, why do you think using the umbrella term OR is appropriate?
First, both the manuscript and the rebuttal appear entirely AI-generated, which seriously undermines my confidence in the credibility of the reported data. Second, the initial submission exceeded the page limit and should have been desk-rejected under ICLR’s submission rules. Thus, I maintain my current score.
I sincerely hope the authors will carefully revise the writing themselves and independently verify all reported data, as this would substantially strengthen the paper’s credibility. I trust the authors understand the concerns I am raising.