acceptodds
Withdrawn from ICLR 2027

Verifier-Native Control and Trajectory Evaluation for Operations Research Repair Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Operations Research (OR) repair is iterative: a candidate edit is solved, checked, diagnosed, and revised. We study how LLM agents use this feedback while preserving operational validity. **Repair-Sufficient State (RSS)** compiles solver and domain-checker observations into typed failure state; **Counterfactual Verifier Loop (CVL)** organizes probes and repairs around verifier transitions; and **OR-INSPECT-BENCH** evaluates fixed-trace diagnosis and live repair. Across 19 directly measured model snapshots, the full pipeline improves strict recovery over one-shot prompting. Matched controls show that RSS improves on raw multi-turn feedback across three anchors. In a fresh same-window Qwen comparison, adding reference-free output normalization to RSS raises strict all-error recovery from 30.6% to 42.8%; the full CVL-r2 configuration reaches 45.8%, a further 3.1-point estimate. A Qwen factorial also finds that canonical-emission correction raises strict recovery by 14.4 and 20.6 points with probes enabled and disabled. Across six paired model anchors, additional control gains vary by model. OR-INSPECT-BENCH combines fixed-trace diagnosis, live repair, and composition holdouts. Together, the results quantify contributions from typed verifier state and output handling, and characterize when added control improves recovery.

Primary area:
optimization
Keywords:
large language models, LLM agents, verifier-guided control, self-correction, operations research, optimization modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r3RxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OR-INSPECT-BENCH, a trajectory-aware benchmark and evaluation harness. It tests LLM agents on multi-turn Operations Research (OR) repair tasks with rich verifier feedback. The tasks cover linear and mixed-integer linear programming formulations. Most prior work studies one-shot translation from text to formulation. The authors study iterative debugging instead. To support this, they propose two interface-level components:

Repair-Sufficient State (RSS). A middleware parser that compiles raw solver logs, domain checks, and constraint conflicts into a structured five-channel state.

Counterfactual Verifier Loop (CVL). An inference-time search controller. It uses UCT and verifier transitions to choose diagnostic probes and candidate repairs.

The authors run experiments across 23 model snapshots, 360 hardened benchmark cases, and five external public benchmark suites. Their component analysis shows how typed verifier feedback and output handling affect agentic recovery.

Critical strengths

1. Shift from one-shot to iterative debugging. Prior benchmarks such as NL4Opt and MAMO test single-pass translation. This paper tests multi-turn error correction with exact solver feedback. That is a real bottleneck in practical optimization modeling.

2. Rigorous and transparent ablations. The authors deserve credit for the matched three-arm studies and factorials in Section 5.2. They isolate the performance gain from output normalization and the gain from search control, and report each separately.

3. Comprehensive evaluation design. OR-INSPECT-BENCH combines static diagnosis (TRACEJUDGE) with live multi-turn correction (REPAIRRUN). Both run under controlled perturbation types and verification gates.

Critical weaknesses

1. The paper’s own controlled experiments show that the headline search controller (CVL-r2) adds only +3.1% over a baseline with a reference-free output normalizer. That gain is not statistically significant; the 95% confidence interval includes zero. The normalizer alone adds +12.2%. In vast majority of cases, the CVL-r2 loop runs a single LLM decision. The search architecture is rarely exercised.

2. The benchmark instances are built from clean textbook problems with five fixed, pre-declared error types injected (e.g. objective flip, missing capacity). The agent’s operator schema matches those same five perturbations. This reduces open-ended mathematical modeling to multiple-choice classification plus slot filling. Once it is constrained into multiple-choice, the problem is much less challenging.

3. The paper leans on proprietary acronyms (RSS, CVL, OR-INSPECT-BENCH, TRACEJUDGE). Some of them obscure standard software engineering and compiler-harness concepts.

Questions for authors

1. Given that adding the reference-free output normalizer to RSS yields a major performance leap (+12.2%) while adding the full CVL-r2 controller yields a statistically insignificant +3.1% gain (and only makes a single decision in 97.5% of cases), how do the authors justify the architectural and computational overhead of the CVL search tree?

2. How do the authors expect agents evaluated on OR-INSPECT-BENCH's closed-world catalog of 5 synthetic perturbations to generalize to open-ended structural formulation errors in real-world industrial optimization tasks?

3. Since this paper covers 2 out of several OR methods, why do you think using the umbrella term OR is appropriate?

Reviewer_kcznOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how LLM agents use solver and domain-checker feedback to repair operations research models. It introduces Repair-Sufficient State (RSS), a typed representation of verifier observations; Counterfactual Verifier Loop (CVL), a controller for probing and selecting repairs; and OR-INSPECT-BENCH, which combines fixed-trace diagnosis with live repair. The experiments include a broad model comparison, matched feedback and controller studies, output-normalization ablations, and acceptance audits. The clearest findings are that typed feedback improves recovery over raw feedback and that output normalization substantially affects strict repair scores.

Critical strengths
  1. The paper addresses a consequential evaluation problem: restoring solver feasibility can still leave the underlying operational model incorrect. Evaluating rationality alongside feasibility, and pairing trace diagnosis with live repair, is a useful direction for studying repair agents.

  2. The component analysis is more informative than the headline comparison against one-shot prompting. In particular, the fresh RSS / RSS+normalizer / CVL-r2 comparison separates a substantial normalization gain from a smaller, unresolved controller margin. The paper also reports negative or inconclusive results, distinguishes reference-assisted projection from reference-free normalization, and accounts for dependence among variants through source-task clustering. These choices make the empirical conclusions easier to assess.

Critical weaknesses
  1. It remains unclear how much of the benchmark measures substantive OR repair rather than translating already-localized failures into canonical actions. Section 3.1 and Appendix B describe source formulations, numerical data, and executable verification. However, the linked public release contains abstract checker observations and repair schemas, without the underlying executable models or replay implementation. In the released instances, checker observations already name the affected field and target; across all 360 rows, their affected-field sets match the required repair-field sets. This does not establish answer leakage, since localized verifier feedback is part of the intended setting, but it makes a deterministic schema-based baseline important for interpreting task difficulty. The semantic audits only partly resolve the broader validity question: the fresh audit leaves 31/80 cases uncertain, and seven strict successes are rejected by the model judge (Appendix G.7). These disagreements are not proof that the scorer is wrong, but they leave the relationship between strict recovery and actual model correctness insufficiently established.

  2. The evidence for the additional value of trajectory-level control is limited once output handling is matched. In the fresh comparison, RSS+normalizer reaches 42.8% and CVL-r2 reaches 45.8%; the 3.1-point difference has confidence intervals containing zero. Moreover, 351/360 CVL-r2 cases use a single LLM decision (Section 5.2; Appendix G.1), so this experiment provides little direct evidence about repeated feedback-conditioned agent decisions, although solver-side interaction can still occur. Table 2 compares RSS without normalization against CVL with normalization, so its larger margins do not isolate control. The earlier screen also has Qwen RSS outperforming full CVL (80.8% versus 61.4%, Table 22). The authors acknowledge these qualifications, but the remaining contribution is better supported as a study of feedback representation and output handling than as evidence for the proposed control mechanism. The composition holdout reuses the source tasks and primitive errors, and the external benchmark results lack matched component controls, so neither closes this gap.

Questions for authors

This paper is 9.5 pages, violating the submission guideline.

2 comments

First, both the manuscript and the rebuttal appear entirely AI-generated, which seriously undermines my confidence in the credibility of the reported data. Second, the initial submission exceeded the page limit and should have been desk-rejected under ICLR’s submission rules. Thus, I maintain my current score.

I sincerely hope the authors will carefully revise the writing themselves and independently verify all reported data, as this would substantially strengthen the paper’s credibility. I trust the authors understand the concerns I am raising.

Dear Area Chair, SAC, and Program Chairs,

We write to report a severe violation of ICLR reviewing standards by Reviewer kczn (1) The reviewer's review and follow-up seems like AI-generated (built from some standard AI-slop words); (2) after we shipped runnable Gurobi code and deterministic baselines resolving all the reviewer's concern, the reviewer ignored every result, indicative of personal spite/retaliation.

Sentence-by-Sentence Proof That the Review Is Totally AI-Generated

Every paragraph uses mechanical LLM templates ("not X, but Y" / "does not establish X, since Y, but makes Z") :

  • "It remains unclear how much of the benchmark measures substantive OR repair rather than translating already-localized failures into canonical actions." (measures X rather than Y)
  • "This does not establish answer leakage, since localized verifier feedback is part of the intended setting, but it makes a deterministic schema-based baseline important for interpreting task difficulty." (does not establish X, since Y, but makes Z)
  • "These disagreements are not proof that the scorer is wrong, but they leave the relationship between strict recovery and actual model correctness insufficiently established." (not proof that X, but leave Y—copying Appendix G.7). **Three more consecutive LLM AI-SLOP expressions
  • "Moreover, 351/360 CVL-r2 cases use a single LLM decision (Section 5.2; Appendix G.1), so this experiment provides little direct evidence about repeated feedback-conditioned agent decisions, although solver-side interaction can still occur." →\to Paraphrases our Appendix G.1 disclosure.
  • "The authors acknowledge these qualifications, but the remaining contribution is better supported as a study of feedback representation and output handling than as evidence for the proposed control mechanism." (acknowledge X, but Y is better supported as A than as B)
  • "The composition holdout reuses the source tasks and primitive errors, and the external benchmark results lack matched component controls, so neither closes this gap." →\to Lifted from our Section 5.4 / Appendix E caveats.
Reviewer_bhXaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how LLMs agents use feedback from solvers and domain checkers to repair operations research models step by step, while keeping their business meaning. The authors introduce three connected parts. RSS turns feedback into five types of structured state. CVL uses changes in verifier states to guide probes, candidate repairs, and search. OR-INSPECT-BENCH combines diagnosis on fixed trajectories with online repair to evaluate the system. The main benchmark has 360 repair cases from 120 source tasks. Across different models, the full system usually performs better than single-round prompting. Matched comparisons also support the value of structured feedback and output normalization. In the new three-way Qwen comparison, the strict repair rates are 30.6% for RSS, 42.8% for RSS with normalization, and 45.8% for full CVL-r2. So the clearest finding is that the structured verifier interface and output processing help.

Critical strengths
  • Repairing an optimization model is more than getting a feasible or optimal result from the solver. The repair also needs to keep the objective direction, integer constraints, capacity limits, and business rules. The paper checks these conditions along with the repair process. This helps find mistakes that a final feasibility score may miss.
  • The authors compare raw feedback, RSS, RSS with normalization, and several CVL settings. They also add a two-factor experiment on probing and output correction. I find the new three-way comparison especially useful, since it separates the gain from normalization from the extra gain of the full controller. This is more convincing than only showing that the full system beats single-round prompting.
  • The paper uses a paired cluster bootstrap that keeps the three variants of each source task together. It also separates model output failures from server failures. In the matched experiments, the experience stores are kept separate and online writing is turned off. These choices helps reduce the chance that experience collected across runs affects the comparison.
  • The authors report cases where the full controller is not the best option. They also point out that some gain intervals include zero and disclose that the old semantic scoring rules accepted too many incorrect repairs. I appreciate these details, since they help readers judge what each part of the system actually adds.
Critical weaknesses
  • Normalization adds 12.2 points, while CVL-r2 adds only 3.1, with an interval including zero. Only 9/360 CVL-r2 cases use multiple LLM decisions, so evidence for feedback-driven iteration is limited. Probing also shows no clear gain.
  • The normalizer fills in repair values, so results mix model judgment, interface following, and rule-based completion. A baseline without an LLM would help separate these effects.
  • Shared invariants cannot independently confirm business correctness. In the LLM audit, 31/80 cases are uncertain and seven strict passes are judged semantically wrong. These results suggests stronger domain validation is needed.
  • Imputed results may look like measured data, while unequal repeat counts may change task weights. Large Qwen RSS differences across batches also need clearer configuration and model-version records.
Questions for authors
  • Could you use the same output normalizer, candidate count, and compute budget across several models, then compare an RSS feedback loop, simple greedy repair, and CVL? Would the extra control still give a gain under these conditions?
  • Could you build tasks that need at least two LLM decisions, where each decision depends on new observations? For successful trajectories, could you show how feedback is used, how the agent backs out of wrong steps, and whether probes help? How would these results change the assessment of CVL?
  • Which fields and repair values can the normalizer fill in automatically? Using only rules, the public schema, and verifier feedback, how many cases can be repaired without calling an LLM?
  • Could the main cross-model figures and tables show only measured results, with scores recalculated on the same set of unique tasks? Could you also give full details about the imputation method, the weights of repeated runs, and configuration changes between batches?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.