- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies the problem of deciding which trajectories should enter the library of experience learners. The authors introduce AdmitOR, an admission procedure that compares models generated by three LLM families on a given instance and newly generated instances with resampled parameters. AdmitOR selects the largest group of candidate models whose objective values agree on all instances across at least two of the families. The value of the group is only returned if the score of the group is large enough to bound the false-discovery rate among accepted values. When combined with an experience learner, AdmitOR achieves the highest or tied downstream accuracy compared to the ground truth, execution success, and majority vote judges on 4 of 5 benchmarks and on both macro and micro averages.
Critical strengths
The paper is well-written, easy to follow, and addresses the important problem of identifying correct optimization models in the absence of ground-truth labels. The proposed method is well-motivated and, importantly, frames admission as control of the false-discovery rate, which comes with guarantees.
Critical weaknesses
The numerical results are not fully convincing: AdmitOR achieves the best downstream performance in Table 2, but the difference in performance is only significant on the macro scale against majority voting among label-free judges. Moreover, Table 3 suggests that the panel with all three families that agree at dominates AdmitOR with both variants of admission rules in admission precision and recall. The paper also relies on the independent resampling of instances, which does not translate to instances with varied sets of binding constraints. This could explain why the target false-discovery rate is not attained in the experiments.
Questions for authors
- Is guaranteed to be feasible in this setting? If not, are problems with an infeasible never certified? Does that affect downstream performance?
- Can this approach handle problems with a weakly informative objective value, or, in the most extreme case, a constant objective as in pure feasibility problems?
- The papers “EquivaMap: Leveraging LLMs for Automatic Equivalence Checking of Optimization Formulations” (Zhai et al., ICML 2025) and "OptiHive: Ensemble Selection for LLM-Based Optimization via Statistical Modeling" (Bouscary & Amin, AAAI 2026) are highly relevant but not cited.
5 comments
Response to Reviewer zRYy (2/2)
Q1: feasibility of . The rule does not assume that is feasible. A certificate requires values from at least two families at , so a problem whose extracted instance is infeasible, or on which fewer than two programs return a value, is never certified, and an agreement formed only on resampled instances admits nothing (Section 3). Every stream problem has a finite labeled optimum, so a report of infeasibility at points to an error in the extraction or in the program, and abstention is the intended outcome. This applies to 45 of the 293 problems outside the development split that reach certification: in 26 no family returns a value at , and in 19 exactly one does.
These abstentions keep wrong values out of the library, because the host's own vote is least reliable on these problems. Majority vote certifies 27 of them, and 11 of its 27 values are wrong (41%), against 27 of its 227 values on the other problems (12%). AdmitOR admits no trajectory from these problems, so none of these wrong values enters its library, which has the highest macro accuracy in Table 2. The verdicts of both judges are in the anonymous repository.
Q2: weakly informative or constant objectives. The rule already uses feasibility as evidence: a candidate that returns no value on an instance where two others return one is excluded from the clique. When the objective is constant, agreement in value reduces to this agreement in feasibility. Comparing solver statuses, with infeasibility as a comparable outcome, extends the rule to pure feasibility problems, and the revised Section 5 describes this extension. On the stream, the certified value is constant across the informative instances in 4 of the 170 certificates, and all 4 are correct.
Related work. Thank you for pointing us to these two papers. We added both to Section 2.
- EquivaMap decides whether two formulations are equivalent. An LLM proposes a mapping between their decision variables, the second formulation is solved, and the mapped solution is checked for feasibility and objective value in the first. It needs a second formulation and returns an equivalence judgment. AdmitOR compares formulations through their values on shared resampled parameters, without a variable mapping, and returns an admission decision.
- OptiHive generates solvers, problem instances and validation tests in one batch, filters faulty components, and uses a statistical model to estimate the true performance of the components and to select a solver. Like AdmitOR, it needs no labels and evaluates candidates on several instances. It selects one solver for each problem, whereas AdmitOR decides whether any value is admitted, abstains otherwise, and targets an error rate on the admitted values. OptiHive's instances are generated by the LLM, whereas AdmitOR resamples the parameter domain of the extracted specification.
The revised Section 2 states our comparison claim for the error budget on admitted values, which neither method targets.
We hope these analyses address your concerns. If so, we would be grateful if you would consider updating your assessment, and we would be glad to provide further details.
Follow-up to Reviewer zRYy
Dear Reviewer zRYy,
Thank you again for your review. As the discussion period is closing, we summarize our answers to your main points.
- Downstream results. On a second host, the comparison with majority vote has the same sign as in Table 2 (79 against 60 paired items).
- Table 3 and resampling. On the solver-verified calibration set, whose texts are complete, the values that the comparison withholds are wrong in 2 of 3 cases at the two-family level, and the calibrated rule withholds 1 of the 2 wrong unanimous values.
- False-discovery rate. All 22 false certificates trace to benchmark texts or labels, so they do not bear on the choice of draws.
- Q1, Q2, and related work. Our reply answers both questions, and Section 2 now discusses EquivaMap and OptiHive.
If these answers address your concerns, we would be grateful if you would consider updating your score. We are glad to provide further details in the remaining time.
Response to Reviewer zRYy (1/2)
Thank you for your careful review and for the specific questions on the downstream results and on the design of the draws.
Strength of the downstream results. AdmitOR is ahead of majority vote on four of the five benchmarks, tied on OptiBench, and behind on none. The paired item counts, items only AdmitOR answers correctly against items only majority vote answers correctly, are:
The preregistered criterion K2 is the macro comparison. Its 95% interval, [+0.87, +6.68], excludes zero, and it also excludes zero on the corrected item set. Over all items, the exact McNemar test gives p=0.10. The item-weighted average gives OptiBench 55% of the weight, and on OptiBench the two libraries tie (28 against 28). On the other four benchmarks, AdmitOR answers 43 items that majority vote misses and misses 24 that it answers. On a second host the comparison has the same sign, 79 against 60 (comment "Additional results: downstream comparison on a current host"). Against execution success, the paired counts (69 and 55) agree with the claim of equal accuracy with a smaller library.
Table 3. Points 1 to 3 of the general response give the explanation:
Independent resampling and the false-discovery rate. The miss of the target does not come from the resampling design. All 22 false certificates trace to the benchmark: in 20 the text does not determine the label, and in 2 the label is wrong and the certified value is correct (Section 4.3). In these problems the three families solve the same extracted specification and agree on every informative instance, and in 16 of the 22 on every sampled instance. More resampling from the same specification cannot show whether that specification represents the source data. Because the error lies in the shared specification in 20 of them and in the label in the other 2, these 22 certificates do not bear on the choice of draws.
The design of the draws does affect coverage, and we agree with your point there. Independent draws make 27% of resampled candidate-instance pairs infeasible. Redrawing with a solvability screen brings 8 of the 57 withheld unanimous problems to three informative instances, all with the base-unanimous value and 7 of them correct (Appendix C). The certified value changes across the informative instances in 166 of the 170 certificates, so the draws move the optimum. Draws chosen to change which constraints bind could increase both detection power and coverage, and the revised Section 5 describes this extension.