acceptodds
Under review as a conference paper at ICLR 2027

Traceability Matters: Towards Faithful Optimization Modeling via Evidence-Chain Supervision

Abstract

Operations Research (OR) modeling, which translates natural-language problem descriptions into solver-executable mathematical formulations, has traditionally relied on domain experts, making it labor-intensive and difficult to scale. Recent work uses large language models (LLMs) to automate this process, but primarily optimizes for code executability and final-answer accuracy, overlooking whether each modeling decision is faithful to the problem statement. Through a failure analysis of state-of-the-art frameworks, we find that nearly 90% of failures arise from such unfaithfulness: modeling elements bound to the wrong entities or conditions (*mapping misalignment*) and stated conditions missing from the formulation (*information omission*). Human experts guard against both errors by repeatedly checking every modeling decision against the source problem text and confirming that no stated condition is left out, an ability we term *traceability awareness*. Inspired by this observation, we propose **TAGO** (**T**raceability-**A**ware **G**uided **O**ptimization), a framework that trains reasoning LLMs to ground each modeling step in explicit textual evidence. TAGO first constructs StepTrace-OR, a dataset that links every modeling step to verbatim evidence from the problem statement, and then trains models with evidence-aligned supervised fine-tuning followed by curriculum reinforcement learning, using a composite reward that targets binding validity and key-evidence recovery. Across four backbones and five OR benchmarks, TAGO consistently outperforms state-of-the-art baselines on both Pass@1 and Pass@8, improving average Pass@1 by up to **8.14** points over the strongest baseline. Further analyses also show that TAGO substantially reduces misalignment and omission errors and produces more credible evidence chains. Our code is available at [https://anonymous.4open.science/r/TAGO_code-245D/](https://anonymous.4open.science/r/TAGO_code-245D/).

Primary area:
generative models
Keywords:
Operation Research

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_5bbkOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper under review considers LLM-based optimization modeling. The authors demonstrate that LLMs do not faithfully output optimization formulations. In particular, they show that models might omit requirements and/or associate values with wrong entries: the code they generate may still be executable and produce an optimal solution, but to not to the correct problem. To address these issues, the paper designs and implements TAGO. The idea behind TAGO is to train models to express an optimization problem through five ‘evidence-annotated’ components: sets, parameters, variables, objective and constraints. The training pipeline relies on a new synthetic training dataset, termed StepTrace-OR, where each optimization problem includes a mathematical formulation whose individual modeling steps are linked to supporting quotations from the problem statement, together with executable Pyomo code. The pipeline then applies evidence-aligned supervised fine tuning followed by RL. The RL rewards capture evidence-expression consistency, evidence coverage, formatting and solver outcomes. Experiments on five optimization modelling benchmarks report improvements over the same backbone baseline models.

Critical strengths
  1. The paper zooms-in into the shortcomings of LLMs for modeling optimization. I liked the breakdown analysis, distinguishing between mapping misalignment and information omission.

  2. The paper designs a full training pipeline to address the identified issues, including the stepTrace-OR trace, mutistage filtering, careful reward design and an elegant combination of SFT and post-training RL. The results in Table 1 show improvements on multiple backbone models (compared to other SLMs derived from these models), and ablation study (Table 2, Sec 4.3) confirms that all components of the solution contribute.

Critical weaknesses

While the paper is fairly well written, I unfortunately see a major issue with its relevance.

  1. Main weakness: Unclear value compared to stronger models. TAGO teaches relatively small Qwen models to decompose the problem, ground each modeling step in source text, and verify coverage. However, the experiments do not really establish that this produces a new capability. Stronger SLMs and frontier reasoning models may already possess much of the underlying source to formulation alignment capabilities. Even if they do not expose this behavior by default, it may be elicited through a structured prompt requiring, e.g., the same five components, supporting quotations, and a final omission check. As far as I can tell, even the comparison with GPT-4o-based agentic systems doesn't use a matched "TAGO-style" prompt and/or verification protocol. Consequently, the reader is left unconvinced whether specialized traceability training is required with the fast evolvement of reasoning LLMs.

  2. Incomplete formulation-level evaluation and unclear benchmark versions. The main Pass@k metric in Sec. 4.1 considers a completion correct if it is feasible and matches the reference objective, although an incorrect formulation may produce the correct objective. Sections 4.4.1 and 4.4.2 provide useful evidence and error analyses, but they do not measure how frequently this occurs among successful outputs (I am aware it's a hard problem!). More importantly, the paper does not seem to specify the exact versions of IndustryOR, MAMO, NL4OPT, and NLP4LP used in the experiments. The reviewer is aware that the original releases contained substantial errors, including incorrect formulations and reference objectives, and several cleaned versions now exist with different corrections, exclusions, and instance counts. This is particularly relevant (and important) for this paper: erroneous benchmark annotations can affect not only objective-based accuracy, but also the correctness of the five formulation components, the validity and completeness of their evidence links, and the classification of omissions and misalignments. Without verified problem statements, formulations, and reference objectives, it is difficult to determine whether the reported traceability improvements reflect better modeling.

Questions for authors

In relation to weakness 1 above,

  1. Can the authors check how does TAGO compare with stronger open-weight models, such as Qwen3.5-9B and GPT-OSS-20B, under the same evaluation protocol? Evaluating these models without additional training would help determine whether TAGO’s gains persist when compared with more capable base models.

  2. Have the authors evaluated frontier LLMs using both a standard optimization-modeling prompt and a 'TAGO-like prompt' (e.g., explicitly requests the five formulation components, supporting evidence for each component, and a final completeness check?

In relation to Weakness 2 above,

  1. Which exact versions of the five evaluation benchmarks were used, and how were their problem statements, formulations, and reference objectives verified?
Reviewer_ho2GOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TAGO trains optimization-modeling LLMs to connect mathematical expressions with verbatim evidence from problem statements. It constructs 82,849 StepTrace-OR records, applies evidence-aligned supervised fine-tuning, and follows with curriculum GRPO using binding-validity, evidence-recovery, format, and solver-outcome rewards. Experiments cover four backbones and five benchmarks, reporting weighted-average pass@1 gains of up to 8.14 points over the strongest baseline. Additional analyses evaluate extracted evidence chains, shared failure cases, problem difficulty, and sampling budgets.

Critical strengths
  • Explicit evidence-expression links make individual modeling decisions inspectable and directly target omissions and entity-binding errors.
  • Improvements across four backbones and five benchmarks support the practical value of the complete training pipeline.
  • The staged filtering separates rule-based validity checks from semantic judgments, improving control over training-data quality.
Critical weaknesses
  • Recovering evidence spans does not establish that every stated condition is implemented in the generated code.
  • The headline failure proportions lack sample counts and dataset coverage, limiting reproducibility and broader interpretation.
  • Older backbones and agentic baselines limit evidence that TAGO's gains extend to more recent systems.
Questions for authors
  1. The Introduction attributes 56.7% and 32.6% of failures to misalignment and omission, but the diagnostic sample is unspecified. Which datasets and methods were analyzed, how many failures were sampled from each, and how were labels assigned and validated?

  2. Equations (3)-(4) evaluate expression support and evidence recovery. Can these rewards remain high when a cited requirement is omitted from code, and are such cases audited?

  3. Table 3 evaluates each method's correct outputs, while Table 4 selects jointly failed instances. Please report common-success evidence scores and full-suite error rates, together with judge identities, manual-audit sizes, and agreement rates.

  4. Table 1 uses Qwen2.5/Qwen3 backbones and GPT-4o-based Reflexion/OptiMUS agents. Could you test a newer backbone of comparable size and compare against recent baselines such as Opt-Verifier and SAC-Opt (both ICML 2026) under matched inference budgets?

  5. Several embedded icons in Figure 1 are pixelated and difficult to distinguish. Please replace them with vector graphics or higher-resolution images and ensure that labels remain legible at normal viewing size.

Reviewer_5aUdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies LLM-based operations-research (OR) modeling, where generated solver code can execute and match the optimum while the formulation still binds a quantity to the wrong entity (mapping misalignment) or silently drops a stated condition (information omission). To make each decision checkable against the source text, TAGO first constructs StepTrace-OR, linking every modeling step to a verbatim evidence span and executable Pyomo code, filtered by deterministic rule checks and LLM alignment judges. It then trains with evidence-aligned SFT followed by easy-to-hard curriculum GRPO, rewarding binding validity, key-evidence recovery, format, and solver outcome. Across five OR benchmarks and four Qwen backbones (with GPT-4o agents as reference), TAGO attains 73.94% weighted Pass@1 with Qwen3-8B and improves strong trained baselines at both Pass@1 and Pass@8; ablations, evidence-quality, error-breakdown, difficulty, and Pass@K analyses attribute the gains to fewer binding and omission errors rather than to better coding alone.

Critical strengths
  1. The paper separates a numerically correct, executable model from a faithful one and splits unfaithfulness into wrong-entity binding and omitted conditions (Sec. 1, Fig. 8, App. B.1), matching each direction with a separate reward. This makes an otherwise implicit property checkable.

  2. On a single instance, an objective/feasibility check can fail to distinguish an erroneous model that still attains the optimum, such as an under-constrained relaxation, so execution feedback alone is limited for diagnosing faithfulness, even though it remains a strong aggregate correctness signal (Sec. 2).

  3. Verbatim authenticity is a deterministic exact-substring/table-cell check before any LLM is consulted, followed by independent solver execution at 1e-4 and only then semantic alignment judges (Sec. 3.1.2-3.1.3, App. A.2-A.3), so a fluent judge cannot rescue a fabricated citation or a non-solving program.

  4. Stages 2 and 4 deliberately distinguish span authenticity from evidence-math alignment, which targets the exact binding error the paper studies rather than conflating citation presence with correctness (Sec. 3.1.3).

  5. Eq. 4 is standard F_beta with beta^2=2 (beta=sqrt(2); recall weighted 2:1, Table 6), while Table 3 correctly uses balanced F1 (beta=1; 249.2052.70/(49.20+52.70)=50.89); removing either traceability term costs Qwen3-8B -5.34/-3.56 Pass@1 (Tables 2/8). I also reproduced the other reported counts (Fig. 8, Tables 4/5, 74,564+8,285=82,849), indicating careful bookkeeping.

  6. Across four open backbones and two GPT-4o agents, TAGO leads the weighted average everywhere (Tables 1/7), and its IndustryOR edge over SAGE widens with difficulty (+5.13/+12.20/+15.00; counts 19/39, 13/41, 6/20; Table 5), as expected if grounding helps most on constraint-rich problems.

Critical weaknesses
  1. TAGO is trained on DeepSeek-v4-pro binding/recovery labels and Tables 3-4 are scored by another prompt of the same family, with only an unspecified manual spot check and no agreement. The results show verbatim, judge-approved spans, not gains in semantic alignment independent of the judge (Sec. 4.4).

  2. There is no source-edit experiment changing one value/entity/operator to see whether exactly the corresponding decision changes, and no evidence BVR/KER predict correctness beyond the solver reward. Such a test is a useful consistency diagnostic, not a semantic-equivalence oracle, since valid reformulations need not change one canonical expression (Sec. 4.4).

  3. Tables 1/7/9 are single point estimates with no seeds, variance, CIs, or paired tests, and the Pass@K estimator/budget is undefined; the margins are not thereby insignificant, but they cannot be assessed, and one cell reverses (EasyLP Pass@1: SAGE 88.96 vs TAGO 86.50). Multi-seed means and a paired per-instance test are needed.

  4. OR-Instruct-3K seeds and optimum-preserving augmentation share an ecosystem with the five widely used benchmarks, yet dedup is only internal to augmentation (App. A.1) and no overlap check appears. This is absence of evidence, not proven leakage, but near-duplicate templates could be matched without faithful binding.

  5. The main SFT contrast changes both schema and target content, and RL ablations keep the augmented data, so the gain from cited spans and BVR/KER is entangled with a rigid five-part structure and added supervision (Tables 2/8). A structure-matched ablation (same schema/data/tokens, no spans or evidence rewards) would separate the two.

Questions for authors

See the weaknesses.

1 comment

Dear Reviewer,

The rebuttal period is nearing its end. We kindly ask you to read the authors' response, join the discussion, and update the review as soon as possible. Your prompt action will help us proceed to the final decision. Thanks.

Best regards, Area Chair

Reviewer_vavcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses optimization programs that execute and return the expected answer while failing to faithfully represent the original natural-language problem. TAGO binds modeling expressions to verbatim source evidence, and combines evidence-aligned SFT with curriculum GRPO using binding validity, key-evidence recovery, format, and solver-outcome rewards.

Critical strengths
  1. Execution and optimum matching do not establish semantic correctness, making the motivation clear and important. Separating verbatim authenticity, evidence–math alignment, and model–code alignment during data cleaning, and using BVR and KER to address misalignment and omission, provides more targeted supervision than a single outcome reward.

  2. On Qwen3-8B, removing BVR/KER costs 5.34/3.56 points; evidence-aligned SFT improves over plain SFT by 4.75 points, followed by a further 12.20-point RL gain. These results support the recipe's effectiveness.

  3. Applying the same post-hoc extractor to TAGO avoids directly comparing its native citations with citation-free baseline outputs.

Critical weaknesses
  1. Table 4 supports an error difference on jointly failed problems but cannot by itself explain accuracy gains on cases where a baseline fails and TAGO succeeds.

  2. Removing BVR or KER individually is informative, but an outcome-reward control retaining the same evidence-SFT initialization, curriculum, and training budget while removing both evidence rewards is missing.

Questions for authors
  1. Does Table 3's advantage persist on jointly correct instances and an audit of all outputs?

  2. With matched data, SFT initialization, curriculum, and budget, what is the gap between outcome-only control and full TAGO?

1 comment

Dear Reviewer,

The rebuttal period is nearing its end. We kindly ask you to read the authors' response, join the discussion, and update the review as soon as possible. Your prompt action will help us proceed to the final decision. Thanks.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.