The stopping rule is not tested as a decision procedure, as the paper states, and the review status behind its stops is unreported. The paper's question (l.46-47) is answered by Def. 1 and two elementary propositions ("set-inclusion facts, not new active-learning bounds", l.633-634; "The proof is the ordering", l.198). Eq. (3) is finite-scenario min-max regret (Kouvelis & Yu, 1997). Stopping once it is at most ε is the stopping test of regret-based elicitation and of decision-region determination (Boutilier et al., 2005; Javdani et al., 2014). Both are cited as "established foundations" (l.85-87), and the paper says its common-bound rule "applies classical principles" (l.88-89), but neither is mapped onto Def. 1. Because the criterion is classical, the case for significance rests on evidence that the rule makes good decisions, and the paper is explicit that it does not test this: its studies are "separate studies, not components of a measured end-to-end system" (l.287-288), and it claims neither prevalence nor an advantage over existing systems (l.44, 82-83, 467-469). I therefore count the missing test against significance, not as a flaw in what is reported. It bears on the paper's own claims only where the abstract states the Study 1 and review results (l.20-22) without the qualifiers that the body attaches to them (l.399-402, 413-421, 707-708; see my second weakness). Three further limits, each acknowledged in the paper, bear on significance. First, Table 1's three methods all verify the same acceptance rule on the same anchors, and no policy baseline is run (nominal-slack skipping, always-ask, ask-when-optima-differ); App. E names the first of these and the right metrics for a confirmatory study (l.790-794). Second, decision adequacy is never scored on a sample of outputs ("Unknown intent or tolerance remains unscored", l.767). Third, the rule needs "an agreed loss limit" (l.15-16), but no tolerance in the studies comes from a user, as the paper states (l.114-115, 703-704, 782-783), and whether the lead example counts as a decision error depends on that choice. Study 1 uses τ=ηmax(1,∣vanchor∣) with η∈{0,0.05} (l.702). Applying the 5% setting to IndustryOR gives τ = 1,075 for B, or 1,200 when anchored at the A optimum, and both exceed the original action's regret of 1,000 under B (4.7% of the B optimum; its regret under A is 0). At that setting the action is decision-adequate for {A,B} without asking, and even for B alone; it is a decision error only at tolerances below 1,000, such as η=0. The 1,000 loss itself is exact (l.253-255); only the Def. 1 verdict depends on the tolerance. Apart from this case, whose reading A the paper does not report as permitted, I found no case where the paper reports both the outcome specific to this rule (delivering because licensed readings share an acceptable action) and a review status for the readings involved. O012 is the only Study 1 case whose alternatives the paper reports as reviewed and accepted as reasonable (l.215), and it cannot stop at either η: 5% of the B optimum is 4,557, and 5% of any plan within the budget stays below 5,040, while ρ = 5,640. So all 15 valid stops come from L001, L050 and L072, whose review status the paper does not report (the IDs appear only at l.699), or from L081 and O000, whose groups include variants that "contradict explicit source conditions" (l.700-701). Those variants "remain negative controls after review" (l.701), so review took place at least there; what the paper does not report is which readings behind the stops were judged licensed (Q1). The only other candidate, App. H's rent rows ("no further question", 6/6/6 in Table 5), scores LLM answer-scope labels on undescribed private data (l.279-280, 505). The rest of the evidence is small and dependent, as the paper says (l.693-695): about 125 recorded DeepSeek calls, one IndustryOR generation per condition (l.309-310), O012 reused across Studies 1, 2, 3 and 5, LeanOpt descriptions screened in advance (l.342-343), and all intent judgments relayed through one collaborator (l.300-303, 503-504). This limit on significance. What the authors say is needed, "new problems, independent review, and user-defined loss limits" (l.469), is the App. E study, which is new work rather than a revision. A small version of it could change my assessment:
- 50 to 100 randomly drawn NL4Opt, IndustryOR or OptiBench items, with two independent annotators marking the licensed readings;
- the fraction of items where the readings change the decision, and where a common acceptable action exists at stated tolerances;
- one or two systems rescored under decision adequacy versus reference matching;
- the stop/ask rule run with LLM-generated readings against the simple policies above, reporting wrong unconditional deliveries and unnecessary questions.
Study 1's agreement appears to follow from its setup, and the review ablations are at ceiling. The abstract says bound reuse "reduced verification solves from fourteen to six with the same fifteen valid stopping decisions" (l.20-21, Table 1), but this appears to follow from the setup as C.2 describes it. Fourteen is the number of models and six the number of cases (l.313-314, 705-706), and τ is "fixed across its interpretation group" (l.702-703). F0 is "the analyst-supplied largest-set relaxation" (l.705). Suppose it is the largest reading of a nested family and L is its optimum, as l.705-708 suggest. Then L=minhvh, and the Prop. 2 test f(xˉ)−L≤τ is equivalent to f(xˉ)−vh≤τ for every h. Identical stops and zero safe-unresolved cases are then guaranteed (both methods run the same 72 feasibility checks), and zero invalid stops simply restates Prop. 2. The full method is the reference that defines validity, so its zero invalid stops and zero safe-unresolved cases hold by definition. Since the anchors are "Fourteen model optima" (l.702), the 14 excluded anchor-generation solves also appear to supply every vh already. The paper calls these families "supplied, favorable" (l.400) and notes that the saving "does not automatically extend to non-nested model families" (l.707-708). If F0 is one of the readings, "favorable" understates "guaranteed". Both qualifiers are correct as far as they go, but the abstract (l.20-21) carries neither. The informative case is a strict relaxation with L<minhvh, where the bound can fail to certify (B.3, l.641-644), and the paper does not report whether any Study 1 case has that form. The review studies are at ceiling:
- every arm retracts the false O012 claim in all four v29 repetitions (l.742);
- targeted self-review scores 11/11 in Table 2, and all 29 completed verdicts are correct (l.417-418);
- every arm is told which claim to check (l.740-741, 744);
- the verdict differences between arms come only from outputs truncated under a 4,096-token cap (l.747, 752-754).
The paper keeps this null rather than claim a benefit (l.420-421), but summarizes it as "no added correction benefit" (l.63-64), "helps some tasks, but not all" (l.411) and, in the abstract, "did not consistently improve later corrections" (l.21-22), none of which mentions the ceiling; "no detectable difference at ceiling" would be more precise. The positive pilot (App. H, Table 5) is five answers from one private family. The facts arm used 36-38% more output tokens over the whole pilot, and the fact sheets are never shown (l.876, 901-902), limits the paper partly states (l.413-414). Of the abstract's three results, the IndustryOR case is therefore the one that depends on neither the Study 1 design nor a ceiling, and it rests on a single generation per condition, as the paper states (l.309-310). The fix has three parts:
- if F0 is one of the readings in each case, say in the abstract and Sec. 6 that the agreement holds by construction;
- add cases where F0 is a strict relaxation (the LP relaxation of integer readings, or a hull of non-nested readings), reporting per case the readings, L−minhvh, the stops, and the solves with and without anchor reuse;
- rerun the review study with untargeted prompts ("find the error, if any"), inputs on which self-review fails, and budgets that avoid truncation.
Rebuttal
Thank you for asking for clearer implementation and evaluation details. We simplified the paper, added a workflow figure, and ran the common-plan search and clarification procedure.
How readings are obtained. Section 3.1 records the supporting text, quantity, unit, scope, corresponding constraint, and added assumption for each modeling choice. Language checks examine choices such as whether capacity applies before sales or only at month-end. The new policy experiment starts with supplied models; the LLM pilots separately test review prompts. We do not yet have a validated automatic procedure that finds all plausible readings. Figure 1 marks this boundary.
How models are checked.The checker handles continuous LPs with fixed variable meanings. It searches for a plan satisfying one model while violating a constraint in the other. Checking both directions supports feasible-set comparison; objective expressions are checked separately. Numerical results are followed by rational certificate checks. Changes in variable meanings require a mapping. This is narrower than general projected-epigraph equivalence, and the revision states that scope explicitly.
How a common plan is found. Section 3.3 first computes each reading’s optimum, then combines the hard constraints and minimizes the largest regret across readings. If the checked result is within tolerance, the plan is delivered. Otherwise, the controlled experiment asks which rule applies and returns a checked optimum for that rule. An unfinished check remains unresolved; one failed candidate does not prove that clarification is necessary.
Validation and baselines. Section 4.3 reports the following comparison:
All 768 runs finish, and final outcomes are rechecked with exact arithmetic. We also check 256 run pairs with identical public inputs but different hidden answers; their behavior is identical before the first question. This evaluates the procedure after candidate models are supplied, not the full natural-language pipeline. OR-Clarify/InterOPT is discussed but has not been run as a baseline.
Data selection. Section 4.2 gives the new selection path: 245 existing ORLM/NL4Opt outputs, 43 eligible models with explicitly continuous variables and extractable affine expressions, and twelve selected using a fixed seed before the new reviews. Selection does not use review correctness. This differs from the earlier 245/244/212/204 scorer study, which is no longer a result in this revision. We report truncations and proposals outside the checker’s variable scope separately; checker rejection is not automatically a semantic error.
Human judgments. We distinguish mathematical checks under stated assumptions, intended rules fixed by experimental construction, and independent semantic judgments. The fixed-output pilot still needs independent semantic labels. Model-reviewer agreement does not replace them. The production example now describes an explicitly stated assumption that differs from a collaborator-reported convention; the record does not establish independent approval of both readings or the timing of that judgment. Independent annotation and natural-language interaction remain the next steps for evaluating the full process.