acceptodds
Under review as a conference paper at ICLR 2027

Admission Without Answers: Label-Free Certification and Experience Learning for LLM-Based Optimization Modeling

Abstract

Agents that learn from experience improve at optimization modeling by storing solved trajectories and reusing them as skills. A wrong trajectory that enters the library can be retrieved again and again, and on a stream of new problems there is no ground-truth answer to decide with. Existing learners admit trajectories by matching known optima or labels, and label-free substitutes such as execution success or agreement at one instance can admit wrong models. We introduce ADMITOR, a label-free admission gate. It generates models from three model families, runs each on the stated problem and on instances with resampled parameters, keeps the largest group of models whose optimal values agree on every instance across families, and applies a threshold fitted on solver-verified problems to accept, abstain, or escalate, with a finite-sample bound on the false-discovery rate among accepted values. Inside a state-of-the-art skill learner, ADMITOR raises candidate-level admission precision to 0.927, against 0.871 for majority vote over the host's own samples and 0.726 for execution success, and its library, the smallest of the four, reaches the highest macro accuracy over five public benchmarks, 58.4 against 54.8 for majority vote. An ablation on the same records shows that the gain comes from the accepted value being external to the learner and unanimous across families; on this stream, resampling never changed an accepted value and only reduced coverage. The false-discovery bound holds on the calibration set but not on the benchmark stream: an audit of every false certificate traces most of them to benchmark texts that omit or round the numbers needed to reproduce the labeled answer, and a label-free check of the extracted numbers against the text flags most of these cases. Code and benchmarks will be released upon acceptance.

Primary area:
optimization
Keywords:
Large language models, Optimization modeling, Label-free verification, Conformal prediction, Selective prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zRYyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the problem of deciding which trajectories should enter the library of experience learners. The authors introduce AdmitOR, an admission procedure that compares models generated by three LLM families on a given instance and newly generated instances with resampled parameters. AdmitOR selects the largest group of candidate models whose objective values agree on all instances across at least two of the families. The value of the group is only returned if the score of the group is large enough to bound the false-discovery rate among accepted values. When combined with an experience learner, AdmitOR achieves the highest or tied downstream accuracy compared to the ground truth, execution success, and majority vote judges on 4 of 5 benchmarks and on both macro and micro averages.

Critical strengths

The paper is well-written, easy to follow, and addresses the important problem of identifying correct optimization models in the absence of ground-truth labels. The proposed method is well-motivated and, importantly, frames admission as control of the false-discovery rate, which comes with guarantees.

Critical weaknesses

The numerical results are not fully convincing: AdmitOR achieves the best downstream performance in Table 2, but the difference in performance is only significant on the macro scale against majority voting among label-free judges. Moreover, Table 3 suggests that the panel with all three families that agree at θ0\theta_0 dominates AdmitOR with both variants of admission rules in admission precision and recall. The paper also relies on the independent resampling of instances, which does not translate to instances with varied sets of binding constraints. This could explain why the target false-discovery rate is not attained in the experiments.

Questions for authors
  • Is θ0\theta_0 guaranteed to be feasible in this setting? If not, are problems with an infeasible θ0\theta_0 never certified? Does that affect downstream performance?
  • Can this approach handle problems with a weakly informative objective value, or, in the most extreme case, a constant objective as in pure feasibility problems?
  • The papers “EquivaMap: Leveraging LLMs for Automatic Equivalence Checking of Optimization Formulations” (Zhai et al., ICML 2025) and "OptiHive: Ensemble Selection for LLM-Based Optimization via Statistical Modeling" (Bouscary & Amin, AAAI 2026) are highly relevant but not cited.
5 comments
AuthorsOpenReview ↗

Response to Reviewer zRYy (1/2)

Thank you for your careful review and for the specific questions on the downstream results and on the design of the draws.

Strength of the downstream results. AdmitOR is ahead of majority vote on four of the five benchmarks, tied on OptiBench, and behind on none. The paired item counts, items only AdmitOR answers correctly against items only majority vote answers correctly, are:

Benchmark (items)Only AdmitOR correctOnly majority vote correct
ComplexOR (18)10
IndustryOR (100)82
Mamo ComplexLP (211)2011
OptMATH (166)1411
OptiBench (605)2828
All items (1,100)7152

The preregistered criterion K2 is the macro comparison. Its 95% interval, [+0.87, +6.68], excludes zero, and it also excludes zero on the corrected item set. Over all items, the exact McNemar test gives p=0.10p = 0.10. The item-weighted average gives OptiBench 55% of the weight, and on OptiBench the two libraries tie (28 against 28). On the other four benchmarks, AdmitOR answers 43 items that majority vote misses and misses 24 that it answers. On a second host the comparison has the same sign, 79 against 60 (comment "Additional results: downstream comparison on a current host"). Against execution success, the paired counts (69 and 55) agree with the claim of equal accuracy with a smaller library.

Table 3. Points 1 to 3 of the general response give the explanation:

  • The panel rows are AdmitOR without resampling, and their precision comes from external verification across families.
  • Resampling can only withhold a value. The values its comparison withholds are wrong in 3 of 7 cases on the stream, against 15.5% of all values on which two families agree at θ0\theta_0, and in 2 of 3 cases on the calibration set, whose texts are complete (comment "Additional results: the comparison on the calibration set").
  • The coverage loss comes from the informativeness requirement, not from the comparison.

Independent resampling and the false-discovery rate. The miss of the target does not come from the resampling design. All 22 false certificates trace to the benchmark: in 20 the text does not determine the label, and in 2 the label is wrong and the certified value is correct (Section 4.3). In these problems the three families solve the same extracted specification and agree on every informative instance, and in 16 of the 22 on every sampled instance. More resampling from the same specification cannot show whether that specification represents the source data. Because the error lies in the shared specification in 20 of them and in the label in the other 2, these 22 certificates do not bear on the choice of draws.

The design of the draws does affect coverage, and we agree with your point there. Independent draws make 27% of resampled candidate-instance pairs infeasible. Redrawing with a solvability screen brings 8 of the 57 withheld unanimous problems to three informative instances, all with the base-unanimous value and 7 of them correct (Appendix C). The certified value changes across the informative instances in 166 of the 170 certificates, so the draws move the optimum. Draws chosen to change which constraints bind could increase both detection power and coverage, and the revised Section 5 describes this extension.

AuthorsOpenReview ↗

Response to Reviewer zRYy (2/2)

Q1: feasibility of θ0\theta_0. The rule does not assume that θ0\theta_0 is feasible. A certificate requires values from at least two families at θ0\theta_0, so a problem whose extracted instance is infeasible, or on which fewer than two programs return a value, is never certified, and an agreement formed only on resampled instances admits nothing (Section 3). Every stream problem has a finite labeled optimum, so a report of infeasibility at θ0\theta_0 points to an error in the extraction or in the program, and abstention is the intended outcome. This applies to 45 of the 293 problems outside the development split that reach certification: in 26 no family returns a value at θ0\theta_0, and in 19 exactly one does.

These abstentions keep wrong values out of the library, because the host's own vote is least reliable on these problems. Majority vote certifies 27 of them, and 11 of its 27 values are wrong (41%), against 27 of its 227 values on the other problems (12%). AdmitOR admits no trajectory from these problems, so none of these wrong values enters its library, which has the highest macro accuracy in Table 2. The verdicts of both judges are in the anonymous repository.

Q2: weakly informative or constant objectives. The rule already uses feasibility as evidence: a candidate that returns no value on an instance where two others return one is excluded from the clique. When the objective is constant, agreement in value reduces to this agreement in feasibility. Comparing solver statuses, with infeasibility as a comparable outcome, extends the rule to pure feasibility problems, and the revised Section 5 describes this extension. On the stream, the certified value is constant across the informative instances in 4 of the 170 certificates, and all 4 are correct.

Related work. Thank you for pointing us to these two papers. We added both to Section 2.

  • EquivaMap decides whether two formulations are equivalent. An LLM proposes a mapping between their decision variables, the second formulation is solved, and the mapped solution is checked for feasibility and objective value in the first. It needs a second formulation and returns an equivalence judgment. AdmitOR compares formulations through their values on shared resampled parameters, without a variable mapping, and returns an admission decision.
  • OptiHive generates solvers, problem instances and validation tests in one batch, filters faulty components, and uses a statistical model to estimate the true performance of the components and to select a solver. Like AdmitOR, it needs no labels and evaluates candidates on several instances. It selects one solver for each problem, whereas AdmitOR decides whether any value is admitted, abstains otherwise, and targets an error rate on the admitted values. OptiHive's instances are generated by the LLM, whereas AdmitOR resamples the parameter domain of the extracted specification.

The revised Section 2 states our comparison claim for the error budget on admitted values, which neither method targets.

We hope these analyses address your concerns. If so, we would be grateful if you would consider updating your assessment, and we would be glad to provide further details.

AuthorsOpenReview ↗

Follow-up to Reviewer zRYy

Dear Reviewer zRYy,

Thank you again for your review. As the discussion period is closing, we summarize our answers to your main points.

  • Downstream results. On a second host, the comparison with majority vote has the same sign as in Table 2 (79 against 60 paired items).
  • Table 3 and resampling. On the solver-verified calibration set, whose texts are complete, the values that the comparison withholds are wrong in 2 of 3 cases at the two-family level, and the calibrated rule withholds 1 of the 2 wrong unanimous values.
  • False-discovery rate. All 22 false certificates trace to benchmark texts or labels, so they do not bear on the choice of draws.
  • Q1, Q2, and related work. Our reply answers both questions, and Section 2 now discusses EquivaMap and OptiHive.

If these answers address your concerns, we would be grateful if you would consider updating your score. We are glad to provide further details in the remaining time.

Reviewer_JEbHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AdmitOR, a label-free admission mechanism for experience learning in LLM-based optimization modeling. It uses multiple LLM families to generate optimization models, compares their optimal values across the original and perturbed problem instances, and applies a calibrated agreement threshold to decide which trajectories can be used for skill distillation. Experiments on five benchmarks show improved admission precision and downstream macro accuracy over majority voting, while also revealing limitations in calibration transfer and parameter resampling.

Critical strengths
  • The paper addresses a practical and relevant problem in experience learning, where incorrect trajectories may contaminate persistent skill libraries when ground-truth answers are unavailable. The proposed label-free admission mechanism provides a reasonable approach to improving the reliability of stored experiences. 
  • The collect-once, replay-many protocol enables controlled comparisons between different admission strategies. The experiments show improved admission precision and downstream macro accuracy over majority voting, providing evidence of the effectiveness of external cross-model verification. 
  • The theoretical analysis provides useful justification for the proposed framework. In particular, the resampling result characterizes when differences between optimization models can be detected, while the calibration analysis explicitly states the assumptions required for controlling false certifications. This helps clarify both the theoretical motivation and the limitations of the method.
Critical weaknesses
  • Table 3 shows that simple three-family unanimity without parameter resampling achieves higher admission precision (0.938 vs. 0.927) and recall (0.760 vs. 0.601) than the full AdmitOR admission rule. As acknowledged in Section 4.2, resampling did not change any accepted value on the evaluated stream and primarily reduced admission coverage. Moreover, the three-family unanimity baseline is not evaluated on downstream tasks, making it difficult to establish the actual benefit of the proposed resampling mechanism. 

  • The method introduces additional test-time computation through multiple LLM calls and solver evaluations, but the paper lacks a systematic compute-performance analysis. It remains unclear how performance scales with the number of model families or resampled instances, whether the fixed computational budget is justified, and how AdmitOR compares with simpler verification strategies under matched budgets. Given the limited observed benefit of resampling, its computational efficiency is an important concern. 

  • The experimental comparisons are largely limited to execution success and majority voting, despite the existence of related work on optimization-model verification and behavioral testing [1, 2]. Without comparisons to these approaches, it is difficult to assess how much AdmitOR advances beyond existing verification techniques. 

  • Although the paper provides a detailed analysis of calibration failures, the false-certificate rate reaches 15.9% on the benchmark stream, exceeding the stated 10% budget. This raises concerns about the practical applicability of the proposed FDR guarantee, especially when the transfer assumption cannot be verified in advance. Additionally, all admission experiments rely on a single training stream and one host learner, leaving the method's generalizability to other experience-learning systems unclear.

  • The reproducibility of the reported experiments is difficult to assess. While the abstract states that code and benchmarks will be released upon acceptance, the introduction and appendix describe the code, experimental logs, and reproduction scripts as already released. However, I could not find an accessible repository link in the submission. This inconsistency is concerning given the extensive implementation-dependent experiments and failure analyses reported in the paper.

[1] Lian J J, Sun Y, Chen H, et al. Reloop: Structured modeling and behavioral verification for reliable llm-based optimization.

[2] Zhang Y, Kang Q, Chen Y, et al. SAC-Opt: Semantic anchors for iterative correction in optimization modeling.

Questions for authors
  • Could the authors provide downstream results for three-family unanimity without parameter resampling? Given the results in Table 3, what concrete benefit does resampling provide beyond simple cross-family consensus? 

  • How does AdmitOR perform under different test-time computational budgets? In particular, how do precision, recall, and downstream accuracy change with the number of model families and resampled instances? Could the authors provide compute-matched comparisons with simpler verification strategies? 

  • How does AdmitOR compare with existing optimization-model verification and behavioral testing approaches [1, 2] when adapted to the same admission setting? 

  • Given the observed 15.9% false-certificate rate on the benchmark stream, how can the proposed FDR guarantee be made practically useful when the transfer assumption may not hold? Are there ways to detect such violations without ground-truth labels?

  • The abstract states that the code and benchmarks will be released upon acceptance, while the introduction and appendix suggest that the artifacts have already been released. Could the authors clarify the current availability of these materials and explain the inconsistency?

5 comments
AuthorsOpenReview ↗

Response to Reviewer JEbH (1/2)

Thank you for the detailed and thoughtful review, and for recognizing the replay protocol and our analysis of resampling and calibration.

W1 and Q1: three-family agreement without resampling. Three-family agreement at θ0\theta_0 is AdmitOR's panel without resampling, so its precision supports the design of the panel (general response, point 1). Resampling adds two things. Where the families split, it separates coincidental agreements. Among the 45 problems on which exactly two families agree, it withholds 6 values, and 3 of the 6 are wrong, including two coincidences on valid parameters (samples 147 and 267, reply to Reviewer 9ZuR). It also provides evidence that the certified programs agree as functions of the parameters, which a library that stores the certified program can use. Where all three families agree, the 22 false certificates of the calibrated rule all come from benchmark texts and labels, which a comparison of programs that share one specification cannot detect (W4 below). On the calibration set, whose texts are complete, the comparison does act at this level: it separates one program in 6 of the 78 unanimous values, 1 of them wrong, and withholds 3 of the 123 two-family values, 2 of them wrong (comment "Additional results: the comparison on the calibration set"). The coverage loss in Table 3 comes from the informativeness requirement, not from the comparison: of the 610 trajectories that two-family agreement at θ0\theta_0 admits, the comparison alone removes 14, 3 of them wrong, and the requirement of three informative instances removes a further 183, 12 of them wrong (general response, point 3). Downstream, on a current host, the AdmitOR library is ahead of the three-family library on both averages with an 11% smaller library, within sampling noise (comment "Additional results: downstream comparison on a current host").

W2 and Q2: compute. Point 4 of the general response lists the judges of the scaling analysis with their model calls, solver runs, certificates and admissions. In summary:

  • One family: one extraction call, one generation call and one solve per problem. Its admission precision is 0.923 to 0.928, against 0.871 for majority vote, which makes no additional call.
  • More families: each added family raises precision, to 0.931 to 0.937 with two and 0.938 with three, and lowers the error rate of the certificates to 13.0% with three.
  • Resampling: no model call and about 10 s of solver time per problem (median 10.25 s for the 15 resampled solves, against a median of 59.5 s for a whole certification). Raising mm from 2 to 5 raises coverage from 152 to 170 certificates, and m=4m = 4 and m=5m = 5 give identical decisions. With the calibrated rule, m=2m = 2 gives the highest precision in the table, 0.942, at recall 0.455.
  • Matched budgets: three-family agreement at θ0\theta_0 and AdmitOR use the same four model calls per problem and differ by 15 solves. The table gives the precision and recall each reaches.

Simpler strategies at lower cost, a perturbation screen on one program and a single-model judge, are compared in our comment "Additional results: two simpler admission rules": neither reaches the precision of three-family agreement (0.938). Downstream accuracy at matched calls is in the comment "Additional results: downstream comparison on a current host".

AuthorsOpenReview ↗

Response to Reviewer JEbH (2/2)

W3 and Q3: optimization-model verification and behavioral testing. Both cited methods act on the generation of a single program:

  • ReLoop perturbs the input parameters of a generated formulation, re-solves it, and checks whether the solution changes the way a correct model should.
  • SAC-Opt compares semantic anchors extracted from the problem with anchors rebuilt from the generated code, and corrects the components on which they disagree.

Either method can be used inside a panel family to produce better candidates, so they complement AdmitOR. AdmitOR acts after generation. It compares programs from different families, issues a value only when they agree, abstains otherwise, and targets an error rate on the admitted values. In the judge swap, the generation pipeline is held fixed across arms so that only the admission rule changes, which is why we did not swap generators. Used alone as an admission rule, a perturbation test with a fixed threshold cannot be both sound and nontrivial (Li & Hai, 2026, cited in Section 3). For this reason AdmitOR compares candidates and selects its threshold on a calibration set. The revised Section 2 states this distinction for both methods. We also ran a perturbation screen on one program as an admission rule on the stored records: on average over the three families it does not raise precision and lowers recall by 28 to 29 points (comment "Additional results: two simpler admission rules").

W4 and Q4: the 15.9% rate, and detecting violations without labels. The rate has a precise cause, and it lies in the benchmark rather than in the gate. All 22 false certificates violate clause (i) of Assumption 1 or carry a wrong label. In 15 the text omits data that determine the label, in 5 the printed values are rounded, and in 2 the label is wrong and the certified value is correct (Section 4.3).

The calibrated threshold behaves on the stream as on the calibration set. It excludes the two-family band because 8 of that band's 39 accepts on the calibration set are false (20.5%), against 2 of 64 three-family accepts (calibration-set counts in the appendix). On the stream, out of sample, the excluded band is wrong in 7 of its 32 value-bearing accepts (21.9%), and the admitted band in 22 of 138, all 22 for the benchmark causes above. The two-family band is unreliable on both sets, and the calibration identifies it from solver-verified problems alone.

Violations of clause (i) can be detected without labels. The numeric-coverage check of Appendix B.1 was designed after the audit and evaluated post hoc on the stored specifications. It flags 9 of the 15 missing-data cases and none of the 116 concordant admissions or of the 148 calibration specifications. Removing the flagged problems leaves 129 certificates. Scoring the two label errors with their corrected labels, 11 of the 129 certified values are wrong (8.5%). These 11 are the 6 missing-data cases that the check does not flag and the 5 rounded texts. No text-only check can detect the rounding, because the printed values are present. When both clauses of Assumption 1 hold, Proposition 2 applies as stated. The restated Proposition 2 gives its 95% statement at δ=0.05\delta = 0.05 without a union bound over the score grid (revision note).

W5: one training stream and one host. Thank you for raising this. The downstream comparison now covers a second host model: on DeepSeek-V4-Flash, the AdmitOR library again has the highest macro and item-weighted accuracy, with the smallest library (comment "Additional results: downstream comparison on a current host"). The admission stream is the OptMATH training set, and the revised Section 5 states this scope.

W6 and Q5: code availability. Thank you for catching this. The abstract sentence was outdated. The code and the stored records are released, as the introduction states, and the revised abstract says so. The anonymous repository, linked in a separate comment, contains the code, the stored verdicts, the four libraries, the 22 case packets, and the analysis scripts with their outputs, including every analysis in this discussion. Its README states for each script whether it runs from the copy alone. The run-record archives, about 0.29 GB, are not part of the copy because of their size.

We hope these answers, the additional experiments and the revision address your concerns. If so, we would be grateful if you would consider updating your assessment, and we would be happy to clarify any point further.

AuthorsOpenReview ↗

Follow-up to Reviewer JEbH

Dear Reviewer JEbH,

Thank you again for your detailed review. As the discussion period is closing, we summarize what we added in response to your questions.

  • Q1. Downstream results for three-family agreement on a current host, and the comparison replayed on the solver-verified calibration set, where it withholds wrong values at both the two-family and the three-family level.
  • Q2 and Q3. The scaling and cost table over families and resampled instances, and two simpler admission rules, a perturbation screen on one program and a single-model judge. Neither reaches the precision of agreement across families.
  • Q4. The cause of each of the 22 false certificates, and the out-of-sample behavior of the calibrated threshold.
  • Q5. The anonymous repository, with the code, the stored records, and every analysis of this discussion.

If these additions address your concerns, we would be grateful if you would consider updating your score. We are glad to clarify any point in the remaining time.

Reviewer_9ZuROpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to decide which trajectories to add to a skill library when reference answers are unavailable. This is important because admitting trajectories solely based on majority voting over the model's own samples may allow incorrect reasoning steps to enter the library and be repeatedly reused.

The authors propose AdmitOR as an admission mechanism for skill libraries. Simply put, three different models generate programs for a given problem, and the program parameters are perturbed to create diverse test instances. When agreement among the models on these instances meets the admission threshold and the programs execute successfully, trajectories matching the accepted objective value are admitted to the library.

The experiments replay existing records from 300 OptMATH training problems with different admission rules, then evaluate the resulting skill libraries on 1,100 problems across five benchmarks. In the main experiment, AdmitOR achieves a candidate-level admission precision of 0.927, compared with 0.871 for majority voting over the host agent's samples.

Critical strengths
  1. The problem is important and practically relevant. Admission into a persistent skill library requires a stronger check than whether a program executes, since incorrect experience can affect many subsequent tasks. Treating admission as a selective decision with an abstention option and an explicit error budget is a useful perspective.

  2. The experimental reporting is transparent. The authors explicitly report the failure to meet the calibration criterion, the limited contribution of resampling, the loss of coverage, and retrieval failures in the ground-truth arm.

  3. The method provides an empirical improvement. The main experiment reuses the same host trajectories and solver records. Compared with majority voting over the host agent's samples, AdmitOR improves macro-average accuracy by 3.53 percentage points.

Critical weaknesses

W1. The core resampling mechanism is not shown to improve admission outcomes.

Table 3 provides the most important comparison. Requiring only that all three model families agree on the original instance achieves an admission precision of 0.938 and a recall of 0.760. Uncalibrated AdmitOR achieves 0.927 and 0.601, respectively; after calibration, these become 0.936 and 0.505. Thus, on the evaluated host trajectories, the simpler original-instance agreement rule achieves both higher precision and higher recall.

W2. The downstream comparison does not separate the contribution of the admission mechanism from that of external multi-model verification.

The majority-vote baseline uses three trajectories generated by the host's single DeepSeek-V3.2 model. AdmitOR obtains its reference objective value from three model families—DeepSeek-V3.2, GPT-5.4, and Claude Sonnet 4.6—with different prompting strategies and solver configurations. Although the host trajectories are held fixed, the model resources used for verification and the source of supervision differ.

Table 3 helps distinguish these factors at admission time, but the multi-model agreement rules based only on the original instance are not evaluated downstream. The paper explains that the host backbone became unavailable during the study. This is a practical limitation, but it still leaves a key comparison incomplete. In particular, the paper does not establish that AdmitOR is better than using the same three models without resampling or calibration.

Questions for authors
  1. Can you use a currently available host agent to evaluate the downstream performance of libraries produced by two-family agreement and three-family unanimity on the original instance? This is the most important missing comparison.
  2. Can you demonstrate a setting where perturbation testing improves admission quality over original-instance agreement using the same model panel? In particular, how can model-level error detections be used when the host does not generate parameterized programs?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 9ZuR

Thank you for your careful review and for the two precise questions.

Q2 and W1: a setting where resampling improves admission, with the same panel. The stream contains it. In samples 147 and 267, two families agree at θ0\theta_0 on a value that the withheld label contradicts: 2927 against 3140, and 3775 against 3898. Agreement of two families at θ0\theta_0 certifies both values. AdmitOR withholds both, because the resampled instances show that the two programs are different functions of the parameters.

  • Sample 147 (unit commitment over four periods). The GPT program limits the generators operating in each period to those operating in the previous period plus those started; the DeepSeek program does not. At θ0\theta_0 the constraint does not change the optimum, and both programs return 2927. On every resampled instance it does, and the values separate.
  • Sample 267 (unit commitment over five periods). The GPT program declares power output integer, as the text states, and the DeepSeek program declares it continuous. At θ0\theta_0 the demands and output limits are integers, so integrality does not change the optimum, and both programs return 3775. On the resampled instances, whose data are fractional, the two values differ by 0.1% to 0.3%, above the agreement tolerance, on all five instances (four draw within the text's domain).

In both problems the program with the extra constraint, or with integer output, returns the same wrong value at θ0\theta_0; their agreement there is a coincidence at one point, the failure that Figure 1a illustrates.

Over the stream, the comparison withholds 7 of the 245 values on which two families agree at θ0\theta_0, and 3 of the 7 are wrong (43%, against 15.5% of all 245). The third wrong value is sample 28 (3 against label 4), where drawn entries of −1-1 separate the DeepSeek and Claude programs. The other four withheld values are correct: in three (samples 20, 93 and 165) the programs that separate differ only in how they read drawn preference entries of −1-1, which the texts give no meaning, and in the fourth (sample 297) a candidate fails on the resampled instances. The revised Section 5 states this limitation.

At the trajectory level, the comparison alone removes 14 of the 610 trajectories that two-family agreement at θ0\theta_0 admits, 3 of them wrong (precision 0.926 to 0.930); the requirement of three informative instances removes a further 183, 12 of them wrong, and sets the recall of the deployed rule (general response, point 3). The calibration set, whose texts are complete, shows the same pattern: the comparison withholds 3 of its 123 two-family values, 2 of them wrong, and separates one program in 6 of 78 unanimous values, 1 of them wrong (comment "Additional results: the comparison on the calibration set").

On the second part of Q2: OptSkills stores trajectories whose code is not parameterized, so in our experiment the model-level evidence decides only which panel candidates certify a value. Storing the certified panel program, or rewriting a host trajectory against the extracted specification, would use it without any change to the gate (revised Section 5).

W2 and Q1: external verification, multi-model verification, and downstream evaluation. The stored records separate the two sources of verification. One external program from the host's own model raises admission precision from 0.871 to 0.923, and agreement of the three families raises it to 0.938 (general response, point 1). The gain over majority vote therefore does not depend on stronger models, and agreement across families adds to it.

Calibration removes the band that is unreliable on both sets: on the calibration set, 8 of the 39 two-family accepts are false (20.5%) against 2 of 64 three-family accepts, and on the stream, out of sample, 7 of the 32 two-family accepts (21.9%). The 22 false certificates among the 138 admitted all come from benchmark texts or labels (reply to Reviewer JEbH, W4).

Thank you for suggesting a current host. The host's model, DeepSeek-V3.2 in non-thinking mode, was withdrawn by its providers during the study (Section 5), so the host of Table 2 cannot be rerun. We ran the comparison you asked for on a current host, DeepSeek-V4-Flash, with libraries distilled from the same admitted trajectories (comment "Additional results: downstream comparison on a current host"; Table 14 of the revision). There, the AdmitOR library is the smallest and has the highest average accuracy, ahead of three-family agreement on four of the five benchmarks; all differences are within sampling noise. The admitted sets explain the closeness: the deployed rule certifies 142 of the 200 problems on which all three families agree at θ0\theta_0, withholds 58, and adds 28 on which exactly two families agree.

We hope these results answer both questions. If they address your concerns, we would be grateful if you would consider updating your assessment.

AuthorsOpenReview ↗

Follow-up to Reviewer 9ZuR

Dear Reviewer 9ZuR,

Thank you again for your review. As the discussion period is closing, we summarize the two results that answer your questions.

Q1. We ran the downstream comparison on a current host, DeepSeek-V4-Flash. The AdmitOR library is the smallest of the three and has the highest macro and item-weighted accuracy. It is ahead of three-family agreement at θ0\theta_0 on four of the five benchmarks, and the differences are within sampling noise (comment "Additional results: downstream comparison on a current host").

Q2. On the stream, the comparison withholds two wrong values on which two families agree at θ0\theta_0 with valid parameters (samples 147 and 267). On the solver-verified calibration set, it withholds 3 of 123 two-family values, 2 of them wrong, and it separates one program in 6 of 78 unanimous values, 1 of them wrong (comment "Additional results: the comparison on the calibration set").

If these results address your concerns, we would be grateful if you would consider updating your score. We are glad to provide any further analysis in the remaining time.

Comments on the paper

9 comments
AuthorsOpenReview ↗

Anonymous code and stored records

To let you check every number in this discussion, we have made the materials available. The code, the stored verdicts of the 298 certification runs, the four skill libraries, the 22 case packets of Section 4.3, and the analysis scripts with their outputs are at https://anonymous.4open.science/r/AdmitOR-anon-8736/. The analyses reported in this discussion are in release/analyses/additional_2026-11/, and those of our four additional-results comments in additional_2026-11b/, additional_2026-11c/, additional_2026-11d/ and additional_2026-11e/. They include one case packet for each of the six problems on which AdmitOR abstains because the values of the agreeing families separate on resampled instances. They also include one for each of the other four unanimous problems on which resampling excluded one candidate (sample 93 is among the six). The README states for each script whether it runs from this copy alone. The run-record archives are not part of the copy because of their size.

AuthorsOpenReview ↗

Revised PDF

We thank the reviewers for their suggestions. The revision adds the analyses of the discussion. The changes are:

  • Section 2: OptiHive, EquivaMap and Learn-then-Test, and the comparison claim stated for the error budget on admitted values.
  • Section 3: Proposition 2 selects the threshold by fixed-sequence testing (Angelopoulos et al., 2021), which needs no union bound over the score grid. The submitted statement used a union bound over a grid of four thresholds. The restated proposition holds over all eight attainable scores of the released calibration file, selects the same τ=33.3\tau = 33.3, and gives its 95% statement at δ=0.05\delta = 0.05. The section also states the feasibility of the stated instance.
  • Section 4.2: the property of one candidate per family, the decomposition of Table 3, the single-family rows of Table 3, where the downstream gain arises, and the measured cost of one certification.
  • Section 5: the scope of the model-level evidence, constant objectives, the domain of the draws, the single training stream, and draws that target binding constraints.
  • Appendix D.2: the scaling and cost table with every pair of families (Table 11), the per-benchmark paired item counts (Table 8), the six abstentions in which the agreeing values separate (Table 10), two simpler admission rules (Table 13), and downstream accuracy on a second host (Table 14). Of the five unanimous problems on which resampling excluded a candidate, three are execution failures and two are disagreements in value. Section 4.2 and Appendix C of the submission listed one of the two among the failures.
  • Appendix D.3: the judges of Table 3 replayed on the calibration set (Table 16), and the false certificates among the two-family accepts that the threshold excludes on the stream.
  • Abstract: the sentence on the code release.

Since the submission, the text has also been revised for clarity, and the numbering has changed. Tables 1 to 3 and Figures 1 and 2 keep their numbers. The calibration table (formerly Table 8) and the figure on false certificates (formerly Figure 4) moved to Section 4.3, where they are Table 4 and Figure 3. Appendix B.1 is now Appendix C, and Appendix C is now Appendix D, with the host reproduction in D.1, the judge swap and its ablation in D.2, and the calibration and the audit in D.3. Former Tables 4, 5, 6, 7 and 9 are now Tables 5, 12, 15, 9 and 6, former Figure 3 is Figure 4, and Tables 7, 8, 10, 11, 13, 14 and 16 are new.

AuthorsOpenReview ↗

Response to all reviewers (1/3): what Table 3 shows

We sincerely thank the three reviewers for their careful reading and detailed comments. All three found the problem important, and two found the framing of admission as a selective decision with an error budget useful (9ZuR, zRYy). The reviews also credit the transparency of the reporting (9ZuR), and the replay protocol and the analysis of resampling and calibration (JEbH).

All three reviews raise one comparison in Table 3. Agreement of the three panel families at the stated instance θ0\theta_0 admits host trajectories with higher precision and higher recall than AdmitOR with resampling. We agree with this reading, and it guided the analyses below. These analyses, computed from the stored records without new model calls, show what it implies for each part of the gate. Their code and outputs are in the anonymous repository.

1. The panel rows of Table 3 are AdmitOR without resampling, and their gain over majority vote comes from external verification, not from stronger models. Two of the three families use models other than the host's, so we evaluated each family alone at θ0\theta_0 (table in point 4). One program from the host's own model, DeepSeek, run through AdmitOR's extraction and generation steps, already raises admission precision from 0.871 to 0.923. The GPT and Claude programs alone reach 0.928 and 0.925. Section 4.2 gives the reason. A wrong modal answer of the host is matched by at least two of its own trajectories, so the 38 wrong certificates of majority vote admit 93 wrong trajectories. A wrong external value is matched only when the host reaches the same wrong value, so the 40 wrong certificates of the DeepSeek program admit 46.

Agreement across families adds to this gain at both levels. Certificates are wrong in 15.5% to 18.4% of cases with one family, 13.2% to 14.5% with two, and 13.0% with three. Admission precision rises from 0.923 to 0.928 with one family to 0.931 to 0.937 with two and 0.938 with three. Outside the 200 problems on which all three families agree, each family alone is wrong in about one third of its certificates: 14 of 40 (DeepSeek), 9 of 26 (GPT) and 22 of 61 (Claude). Inside them, 26 of 200 are wrong, and Fisher's exact test gives p<0.01p < 0.01 for each family. Table 3 therefore identifies the panel, external programs from several families, as the source of the gain in admission precision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.