acceptodds
Under review as a conference paper at ICLR 2027

IDOL: Answer-Free Behavioral Constraints for Small Language Models

Abstract

Behavioral constraints can guide a small language model without supplying a target response, but their contribution can be conflated with changes in prompting, candidate generation, and evaluation. We introduce IDOL, a source-labeled framework for measuring the incremental value of answer-free behavioral guidance. Typed IDOL-DSL programs distinguish requirements recoverable from the prompt from additional behavioral proposals, and compile into deterministic checks and semantic verifiers over student-generated responses. We evaluate canonical and prompt-derived programs through separate generation interventions and fixed-pool selector comparisons, using two automated judges in both response orders. On a fixed, preselected 300-prompt UltraChat-derived panel across three generation seeds, canonical guidance improves automated scores over a generic sham under both judges, while the closest explicit and permuted controls do not establish an advantage. Accepted latent constraints occur in 66 prompts and change 28 of 900 shared-pool selections relative to prompt-derived scoring. The targeted dual-order selector estimate is under DeepSeek, with a 95% interval of , and under Phi, with : both point estimates are positive, but the declared cross-judge support criterion is unmet. Controlled, filtered-rationale, and benchmark extensions yield inconclusive or mixed results. IDOL provides an executable representation and an attribution-focused study of constraint-guided inference, distinguishing a prompting gain from the incremental effect of additional scoring constraints.

Primary area:
foundation or frontier models, including LLMs
Keywords:
small language models, answer-free behavioral constraints, constraint-guided inference, programmatic verification, fixed-pool candidate selection, instruction following, automated evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nMRkOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies answer-free behavioral guidance for small language models through IDOL, a typed, source-labeled constraint representation compiled into deterministic checks and semantic verifiers. It distinguishes prompt-derived requirements from additional behavioral constraints, and separates interventions that change candidate generation from interventions that rerank the same student-generated candidate pool. The main evaluation uses Qwen3-0.6B on a fixed, post hoc panel of 300 UltraChat-derived prompts with three seeds and eight candidates per pool, with two automated judges evaluating both response orders. Canonical guidance improves scores relative to a generic sham, but an advantage over explicit or permuted guidance is not established. Adding latent constraints changes 28 of 900 fixed-pool selections; its aggregate score improvement is positive under both judges but does not meet the declared cross-judge support criterion. Controlled, filtered-rationale, and objective-benchmark extensions give mixed or inconclusive results. The contribution is therefore an inspectable constraint interface and an attribution-focused empirical study, rather than a demonstrated general improvement in small-model performance.

Critical strengths

The separation between generation interventions and fixed-pool selector interventions is a useful experimental design choice (Sections 3-5). Comparing the prompt-derived and combined scorers on identical candidate strings directly isolates the operational effect of the additional constraints, avoiding attribution of candidate-generation changes to the selector.

The paper reports negative and inconclusive results transparently, including limited selection changes, response-order sensitivity, lack of cross-judge support, and mixed benchmark outcomes (Section 6 and Appendices G-I). This makes the empirical scope easier to assess and provides useful cautions for evaluating constraint-guided reranking.

Critical weaknesses

Missing usable citations and bibliography. The submitted PDF contains unresolved citation markers throughout the text, while the References section is empty. Consequently, the discussion of related work cannot be checked, and the claimed contribution cannot be adequately positioned against prior constraint-guided inference, verification, and candidate-selection methods. A complete bibliography and properly resolved in-text citations are needed to assess the paper's originality and context.

The practical value of the full method is not established. A central design question is why a large model should assist a small model at inference time rather than directly produce the answer itself. For configurations requiring such assistance, the paper needs to justify what practical benefit the additional IDOL pipeline provides. The paper does not directly compare a large language model answering each request once with a small language model using the complete IDOL pipeline. The reported comparisons among small-model variants and the cached, model-resident timing measurements in Appendix K do not answer this practical choice. Constraint preparation, multiple candidate generations, verification, and selection introduce additional work that may offset the advantage of using a small model. A head-to-head experiment on the same tasks should compare answer quality, total end-to-end time, and total inference cost for the two alternatives, covering all model calls needed to produce the final answer. The authors should distinguish reusable offline preparation from per-request inference-time assistance. Without this comparison and a clear justification of the large model's role, it remains unclear when IDOL is preferable to simply using the large model directly. This missing evidence is central to my rejection recommendation.

Questions for authors

Please provide a complete reference list and resolve the missing in-text citations, so that the relationship to prior constraint-guided inference, verification, and reranking methods can be evaluated.

Why use a large model to assist a small model at inference time instead of asking that large model to answer the request directly? Please clarify which large-model operations are required for each new request and which can be completed offline and reused. What concrete practical value does the additional IDOL pipeline provide? Please substantiate this design choice with a direct comparison on the same tasks between (a) one answer generated directly by the large model and (b) the final answer produced by the small model through the complete IDOL pipeline. Compare answer quality, total end-to-end time, and total inference cost. Include constraint preparation, candidate generation, verification, and selection wherever required by the evaluated pipeline; identify reusable setup costs separately and state the models and hardware used.

As a complementary ablation, could you evaluate a version in which all model-based stages used to produce the final answer use only small models, including constraint preparation and semantic verification wherever applicable, while retaining the deterministic checks? Compare this small-model-only pipeline with the current IDOL pipeline in answer quality, total end-to-end time, and total inference cost. This would help establish whether large-model assistance is necessary and what value it adds. The reported changes of the candidate-generation backbone do not appear to test this full-pipeline replacement.

3 comments
AuthorsOpenReview ↗

References restored; the requested comparisons give no reason to prefer the pipeline over a direct 7B answer

Thank you for the valuable feedback. We ran both comparisons you asked for; the results do not favor the pipeline. Page, section and table numbers refer to the revised PDF (35 pages); changes are in blue. Your "Appendix K" (latency) is now Appendix L (p. 28).

W1, Q1. References. The submitted PDF was built without its bibliography; the revision resolves every citation (56 references, pp. 11–15). §2 (p. 2), unchanged apart from its citations, covers verifier-based candidate selection, ACT, RLCF, AutoIF and VerIF.

W2, Q2. Pipeline versus a direct answer. Our results give no reason to prefer the pipeline. For a new request, no tested pipeline configuration used less mean GPU time than one direct 7B answer and none scored reliably higher; direct scored higher on GSM8K and on 128 additional IFEval prompts. Reuse for an identical request lowers cost and leaves answers unchanged. We claim no deployment advantage; the contribution is the attribution study (§9, p. 9).

Setup (§6.6, Table 5, pp. 7–8; Appendix K, pp. 26–28). Direct: one DeepSeek-R1-Distill-Qwen-7B answer (reasoning on, 2,048-token cap). Assisted: per request the 7B model writes one to three checks (the "sketch") and labels each eligible candidate pass/fail on each scored semantic check; Qwen3-0.6B fills 16 candidate slots (512-token cap, thinking off; eligible = complete and passing hygiene checks). All-small: Qwen3-0.6B does every model step. The Qwen3-32B-AWQ labeler of the 300-prompt panel (§5.2, p. 5) was not run. All arms ran each prompt on the same A100 80 GB, twice, one request at a time. Scores use the first run; a request with no eligible candidate counts as a failure (assisted: two original IFEval prompts; all-small: one). GPU seconds (time the GPU is reserved, all stages) average all attempts; a fresh pipeline request used 13–19 times the tokens of a direct answer. Number pairs read assisted, then all-small.

OutcomeArmCountp50/p95 (s)GPU s, freshGPU s, sketch reused
GSM8K correct, of 64Direct 7B607.05/18.168.24–
Assisted911.70/17.9312.112.62
All-small109.25/16.7910.153.09
IFEval strict passes, of 64Direct 7B3610.39/23.5212.10–
Assisted3713.75/34.2617.548.64
All-small4011.70/44.9216.058.45
IFEval strict passes, of 128 additionalDirect 7B8310.62/33.6913.05–
Assisted6813.52/34.7015.65not run
All-small6810.20/37.5513.72not run

IFEval. Intervals are paired 95% bootstraps over prompts, exact for pass rates. On the 128 additional prompts each pipeline is 11.7 percentage points below direct [−21.9, −1.6]; on the original 64, all-small is 6.3 percentage points above [−6.3, +18.8] (loose: 41 vs. 42 for direct). All-small has the lowest median latency on the 128, yet direct was faster on 78 of them (median paired gap 1.06 s [0.3, 1.6]). IFEval had no 0.6B baseline without the pipeline and no thinking-on run.

Reuse. Only the sketch can be prepared ahead, and only for an identical request; candidates are generated and verified every time. Writing the sketch took 47–78% of fresh pipeline GPU time (7.05–9.47 s in the Table 5 runs; constrained decoding at 23–25 tokens/s; direct: about 50). A request that reuses its sketch returns the same answer (fixed seeds). In the Table 5 setting it costs less than a direct answer (table); with thinking on and four candidates (GSM8K, 30 prompts) it costs more, 20.41 s and 18.49 s against 7.42 s (Appendix K, p. 27). Each paired 95% interval excludes zero.

GSM8K (Appendix K, pp. 26–27). With thinking off, a correct eligible candidate existed for 28 and 29 of 64 (36 in a plain-prompt pool without the sketch). Enabling candidate thinking raised this to 57 and 60 and accuracy to 49 of 64 in both arms, at 35.84 and 30.89 mean GPU seconds: 17.2 percentage points below direct [−29.7, −4.7]. With thinking on, the selector solved 46–49 of 64 at N = 4, 8, 16, within one item of the first eligible candidate (45–50) and 2–6 items below a plurality vote (50–52; paired interval excludes zero in 2 of 6 cells) on the same pools (Table 17). In every GSM8K setting it returned the first candidate slot in 72–88% of requests. One Qwen3-0.6B draw with thinking and no pipeline (2,048-token cap) solved 51 of 64.

Q3. All-small pipeline. 7B assistance gave no measurable gain (9 vs. 10, 37 vs. 40, 68 vs. 68; every paired interval includes zero, [−7.0, +7.0] percentage points on the 128) and used 1.5–2.0 s more mean GPU time per fresh request (interval includes zero on the original IFEval prompts). This does not show that the 0.6B model can prepare request-specific constraints: on GSM8K, 44 of its 75 checks carry the same generic text, "brief abstract reason" (7B: 0 of 101; Appendix K, p. 26). Deterministic checks were kept but rare (0.00–0.20 per request).

Thank you for the detailed additional experiments and thoughtful responses. I appreciate the depth of your analysis and your candid assessment of the method’s limitations. However, the current findings still offer limited evidence of the approach’s practical value, and I remain concerned about the significance of the paper’s contribution. In recognition of the thorough investigation and clarifications, I will raise my score.

AuthorsOpenReview ↗

Thank you for the careful reassessment

Thank you for taking the time to examine the additional experiments and reconsider your assessment. Your request for a complete end-to-end comparison helped us clarify both the practical limits and the contribution of the study.

The revised paper uses identical candidate pools to isolate what the added constraints change, while reporting the observed quality and cost trade-offs explicitly. We take your remaining concern about practical value seriously.

We appreciate your thoughtful engagement throughout the discussion and the care you put into evaluating the additional evidence.

Reviewer_w7vGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents IDOL, a method to measure how much extra help small language models get from behavior rules without the answers. It tests two things: (1) changing the instructions used to generate answers, and (2) changing how answers are scored and selected from a fixed set. The solution beat a fake "be careful" prompt, but did not perform better than simply stating the prompt's rules, or than a shuffled version.

Critical strengths

(1) Transparently report the null and negative results.

(2) Separate and compare the impact of changes between prompting and evaluation criteria.

Critical weaknesses

(1) The paper is very difficult to read. For example,

  • What does "canonical guidance" mean? What is the relationship among "canonical programs", "canonical guidance", "canonical discipline" and "canonical library"? There is no place to provide the formulation of the generation condition, prompt, and result.
  • The reference is broken (all display ??). The authors claim the citations are reviewed, but there is no reference in the paper.
  • The information in Figure 1 and Table 1 is very limited. No solid examples of constraints or results.

(2) Lacks a core theme, and the narrative is disorganized. The paper presents IDOL as the constraint framework for small language models in the title and abstract. However, the context is attribution and measurement, and most of the main results are null.

Questions for authors

(1) What does the canonical guidance exactly mean?

(2) How are the latent constraints assigned to the prompt?

(3) How are synthetic profiles created, and why do you have to involve the data?

(4) There is no teacher in the main experiment. Why do you involve the student in the experiment? It is confusing and may be referred to as a distillation paper.

3 comments
AuthorsOpenReview ↗

Canonical guidance defined, references restored, and the paper reframed as an attribution study

Thanks for the constructive feedback! Page, section and table numbers refer to the revised 35-page PDF; changed passages are printed in blue.

W1 (terms), Q1. Canonical guidance ("canonical discipline" in the submitted §4.3) is the generation condition that appends to the explicit prompt (the request plus its profile's requirements) one fixed rule text per accepted category tag (§4.1, p. 3). The canonical library holds four such texts, one per category (Table 9, p. 19); the reasoning text is "Maintain arithmetic and logical consistency without adding unstated facts." A canonical program, now defined in §4.3 (p. 4), is one request's accepted rules from that library, typed in IDOL-DSL; compiled, these rules are the checks the combined selector adds. The submitted statement that the library "contains 68 constraints" was wrong: 68 counts rule instances across the 66 tagged prompts (§4.1, p. 3; Appendix B, pp. 16–17). §3 (pp. 2–3) formalizes candidate pools and scores per condition g.

Conditions, prompt additions and results of the new closed-world control (Table 24, p. 34): 128 tasks with reference answers, categories assigned by construction; Qwen3-0.6B, 3 seeds, 8 draws each; pass = plain JSON, exactly the requested keys in order, correct values.

ConditionAdded to the explicit promptFirst draw (of 384)Any of 8 (of 384)All draws (of 3,072)
Explicit-onlynothing95183 (2.7%)
Matching rulerule for the task's category145992 (3.0%)
Generic caution (sham)"Be careful and check your answer before responding."1599160 (5.2%)
Permuted rulerule for another category106181 (2.6%)

The matching rule shows no reliable advantage (differences in percentage points; Table 25, p. 34). On first draws it differs from explicit-only by +1.3 [−0.8, +3.4] and from permuted by +1.0 [−1.3, +3.1] (simultaneous 95% intervals); over eight draws all four intervals against them include zero, and against the generic caution it is −2.2 [−3.1, −1.4] over all draws and −10.4 [−14.6, −6.3] in any-of-eight (paired 95% intervals). Per-draw pass rates are near floor (at most 5.2%; p. 35), and 34 of the 40-case any-of-eight gap to the caution is in structure tasks (8, 2, 36, 31 of 96 in table order; Table 26, p. 34).

W1 (references). You are right on both points: the submitted PDF contained no bibliography, so no citation resolved, and its statement that an author had checked the citations was wrong for that PDF. The revised PDF prints all 56 references (pp. 11–15), and its AI-use statement (p. 10) no longer lists citations among the author's checks.

W1 (examples). Figure 1 and Table 1 still hold no example. The revision adds the rule and caution texts (Table 9, p. 19) and a paraphrased request with each condition's addition (§4.3, pp. 3–4). For results, §7 (p. 9) paraphrases the changed selections for two requests. No complete generation prompt or answer is printed.

W2. We agree: the submitted title and abstract presented a constraint framework, while the content is an attribution study with mostly null results. The revised abstract and introduction (pp. 1–2) are organized around whether a changed answer comes from the generation prompt or the selector, and the conclusion says the study does not establish a benefit of the added constraints (p. 9). The title is unchanged. The only generation contrast whose intervals exclude zero under both judges is canonical over the generic caution: +0.1794 [+0.0422, +0.3150] (DeepSeek) and +0.2356 [+0.0933, +0.3850] (Phi) points on the automated judges' 1–5 scale (95% intervals; 300 prompts, 3 seeds; Table 2, p. 6). It does not show a benefit of the rule texts: on 234 of the 300 prompts the canonical input is the explicit input, so it mostly compares the explicit prompt with and without the caution, and the 66 tagged prompts alone do not retain a robust positive result (§6.1, p. 6; no estimate is printed).

Q2. Offline, the source labeler proposes category tags for each request (§5.2, p. 5). A tag is accepted if the compiler supports its type and it passes the lexical answer-leakage screens (§4.1–4.2, p. 3). No human has validated the tags (Appendix E.3, p. 20), and the closed-world control fixes categories by construction, so it does not test this step (Appendix U, p. 35).

Q3. We drafted the five fixed profiles with AI help and assigned them by keyword rules on the request text. They give every condition the same explicit, checkable requirements (for decision support: pros, cons and unknowns; §5.1, p. 5), which the prompt-derived checks test (Appendix H.3, p. 24).

Q4. Correct: the main experiment has no teacher, and no language model in the paper is trained or distilled (§2, p. 2). "Student" meant the Qwen3-0.6B that samples the candidates; it is now the "candidate generator" (§3, p. 2; §5.2, p. 5).

Thank you for the thorough revision. The restored bibliography, the corrected AI-use statement, the control definitions, and the corrected 68-instance statement resolve those concerns. However, the clarifications narrow the paper’s positive evidence, and the new experiments add no support. I keep my rating of 1.

  1. No supported benefit remains. The paper now states that canonical > generic caution mostly reflects the appended caution, since canonical input equals explicit input on 234/300 prompts (p.6, Table 6). It also states that it “does not establish a benefit of the added constraints” (p.9). These results come from an outcome-informed panel (pp.4, 16) scored only by automated judges.

  2. The remaining selector effect is fragile. Only 28/900 selections change. Both judges favor the combined selection in both orders on just 7 of them, and 16 are mixed or unresolved (p.9). No reference answer or human label exists for any of them (p.24). Two DeepSeek judging runs over identical strings give different deltas on 12 of the 28 pairs (p.22), and the Phi interval includes zero (Table 2).

  3. Simple same-pool baselines match or beat the selector. In the end-to-end comparison with thinking on, the selector is within one item of the first eligible candidate and 2–6 items below a plurality vote (Table 17). With thinking off, the sketch lowers correct-candidate coverage from 36/64 to 28–29/64 (p.27).

  4. The constraint layer is thin. It consists of four fixed sentences (Table 9) with unvalidated labels (p.20), and there is no comparison with existing selectors such as reward-model best-of-N.

AuthorsOpenReview ↗

How reward models and a plurality vote select from the same answers

Thank you for suggesting reward-model best-of-N. We reranked the thinking-enabled GSM8K candidates from Table 17 (p. 27). For each of 64 questions, the 0.6B model had generated up to 16 candidates with either a 0.6B or a 7B sketch writer. Within each condition, every selector saw the same eligible answers. Each Skywork reward model scored the question and one answer at a time, without the reference answer.

Correct out of 64First eligibleNumeric pluralityIDOLSkywork 1.7BSkywork 4B
0.6B sketch writer5051494044
7B sketch writer5052493440

The 4B reward model scored 10.94 percentage points below IDOL averaged across the two conditions (descriptive 95% paired bootstrap interval −19.53 to −2.34, resampling the 64 questions). Plurality scored highest in both. Across the 15 question–condition pairs where IDOL and plurality differed in correctness, ten favored plurality and five favored IDOL. Each had a unique most frequent numeric answer. The sketch can also make the candidates worse. With thinking off and 16 candidates, the plain prompt yielded a correct eligible answer on 36/64 questions. With a sketch, coverage fell to 29/64 for the 0.6B writer and 28/64 for the 7B writer (Table 17).

Your concern about the four rule texts remains. On our free-form panel, 234 of 300 prompts received no added rule, and we chose to use the panel after inspecting preliminary results (pp. 4–6). The selector changed the answer in 28 of 900 comparisons. Those 28 have no independent correctness labels, and two DeepSeek judging runs gave different score margins to 12 pairs even though they saw the same answers (p. 22). The four categories are heuristics. Even on 128 tasks where we supplied the category, adding the matching rule changed the first-draw pass rate by +1.30 points over explicit instructions alone (97.5% interval −0.78 to +3.39). Over all draws, a generic caution produced 160 passes out of 3,072, versus 92 for the matching rule (Tables 24–25, p. 34). We appreciate the distinction you drew here. On a task with checkable answers we can measure the effect of choosing a selector. A reliable benefit from the four rules on natural requests remains unestablished.

Reviewer_LzYLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The IDOL study investigates how to guide small language models to generate and select responses through behavioral constraints without providing target answers or updating model parameters. The authors propose the Constrained Language IDOL-DSL, which distinguishes between requirements already present in the prompt and additional behavioral norms, and converts them into rule-checking or semantic verification. The main contribution is the establishment of a verifiable constraint enforcement framework, which uses a controlled design with a fixed pool of candidates to separately measure the effects of prompt guidance and additional constraint filtering. In terms of validation, the paper uses Qwen3-0.6B as the primary model, conducts a two-judge evaluation across 300 prompts and three random seeds, and performs extended experiments on IFEval, IFBench, and GSM8K. The results show that normalized guidance improves automated scoring compared to general prompts, and that additional constraints also alter the selection of final responses in some samples.

Critical strengths

The paper’s key contribution lies in its separate analysis of the generative changes induced by prompts and the filtering effects resulting from additional constraints, using a shared candidate pool for comparison, which provides a clearer attribution of performance changes. Additionally, IDOL-DSL explicitly records the source, type, and verification method of constraints; combined with execution logs, this enhances the verifiability and traceability of the constraint application process, providing a practical analytical framework for studying how behavioral constraints influence the responses of small models.

Critical weaknesses

Existing experiments have not yet fully validated the consistent benefits of additional constraints: compared to the control groups using explicit and randomized prompts, no clear advantage has been established; nor have the gains achieved on a fixed candidate pool met the standard for unanimous support from both judges. The main experiment relied on 300 prompts selected post hoc; the positive evaluations lacked human verification, and the results from objective benchmarks and cross-model generalization did not provide consistent support, limiting the assessment of the method’s practical value and generalization ability. Furthermore, on page 10 of the current PDF, there is only a “References” heading, followed immediately by the appendices, while citations in the main text are displayed as “?” or “??” in many places. The absence of reference entries makes it difficult to verify related work, method origins, and novelty, constituting a substantive issue that affects the paper’s completeness and evaluability.

Questions for authors

What are the specific screening criteria for the 300 prompts in the main experiment? Were the model’s outputs or evaluation results taken into account during the screening process? Regarding the 28 instances where candidate selections were actually modified due to additional constraints, could you explain—in light of previous answers—which verifiable improvements the positive scores primarily correspond to, such as factual accuracy, adherence to instructions, or changes in phrasing? Is the absence of a reference list and the display of question marks in the in-text citations a compilation error? Please complete the citations so that we can verify the relationship to existing work such as ACT and RLCF, as well as the scope of your contribution.

1 comment
AuthorsOpenReview ↗

No rule-specific benefit in a new closed-world control; panel screen and 28 switches detailed; references restored

Thank you for the constructive feedback! W1–W6 and Q1–Q3 follow your order. W1–W5 still stand; W6 is fixed. Pages refer to the revised 35-page PDF; changes are in blue. Intervals are 95%.

W1. Controls. Correct: on the public panel both control intervals (explicit, permuted) include zero (DeepSeek; Phi not reported; Table 2, p. 6). Canonical guidance scores above the generic caution by +0.1794 [+0.0422, +0.3150] (DeepSeek) and +0.2356 [+0.0933, +0.3850] (Phi) on the 1–5 scale (300 prompts × 3 seeds), but its input equals the explicit input on 234 of 300 prompts (§4.1, p. 3), so this mostly tests the appended caution, not the rule texts, and the 66 tagged prompts alone retain no robust positive result (§6.1, p. 6; no estimate printed).

New closed-world control (Appendix U, pp. 34–35): 128 tasks with reference answers, categories assigned by construction; Qwen3-0.6B, 3 seeds, 8 draws. Pass = plain JSON, exactly the requested keys in order, correct values.

PassesExplicit-onlyMatching ruleGeneric cautionPermuted rule
First draw, of 3849141510
All draws, of 3,072839216081
Any of eight, of 38451599961
Any of eight, structure, of 96823631

On first draws the matching rule differs by +1.3 percentage points (pp) [−0.8, +3.4] from explicit-only and +1.0 pp [−1.3, +3.1] from permuted (simultaneous; Table 25, p. 34). Over eight draws no difference from either is detected (four intervals include zero); it is below the generic caution: −2.2 pp [−3.1, −1.4] over all draws and −10.4 pp [−14.6, −6.3] in any-of-eight. Pass rates are near floor; no rule specificity is shown, and the judge-score advantage over the caution does not carry over to these tasks (§6.1, p. 6).

W2. Two-judge criterion. Still unmet: combined minus prompt-derived selection is +0.0239 [+0.0033, +0.0583] (DeepSeek) but +0.0144 [−0.0067, +0.0500] (Phi), on the 1–5 scale over 900 pools (Table 2, p. 6). These come from the targeted judging run, a post-inspection audit (§5.3, p. 6); a separate broad run of the same pairs shows the same pattern (Appendix G.1, p. 22).

W3, Q1. Panel. Yes, it is post hoc: assembled after the original study and not preregistered, so its results are exploratory (§5.1, p. 5; Appendix B, p. 16). Before the factorial, a text screen scanned 3,956 first user turns of an UltraChat shard, kept those of 8–160 words, removed duplicates, excluded requests such as code-heavy, transformation and safety-sensitive ones, and chose 60 per category by keyword rules (reasoning, evidence, decision support, planning, uncertainty); the keyword and exclusion lists are not in the PDF (§5.1, pp. 4–5). The screen read request text only, but we adopted the panel only after a headroom pilot, the primary run and judge checks, so its use was outcome-informed.

W4, Q2. Human verification; 28 switches. No human judgments were collected (§5.2, p. 5), so no positive score is human-verified. For the 28 changed selections (of 900; 21 prompts) we cannot attribute the scores to accuracy, adherence or phrasing: no reference answers or labels exist (Appendix H.3, p. 24).

  • Adherence: no measured gain. The replayed format checks are prompt-derived, so both selectors already score them; both answers pass in 22 pairs, only the prompt-derived one in 1, and 5 have none.
  • Judges (broad run): in both orders, both favor the combined answer on 7 pairs and the prompt-derived one on 1, both tie on 4, and 16 are mixed or unresolved; the targeted run gives 6/1/2/19 (§7, p. 9).
  • Two of the 7 are one burger request in two seeds: the prompt-derived answer only copies the request's first question; the combined one gives a preference and a reason. No other improvement is checkable without a judge.

W5. Benchmarks, other models. Agreed: five of six intervals in Table 4 (p. 8) include zero, both other generators among them, and the three-seed, 300-item panel does not corroborate the single-seed GSM8K gain (+2.73 pp [+1.14, +4.32]; 1,319 items; §6.5, p. 7). One direct 7B answer also beat both IDOL pipelines on GSM8K (60 against 9 and 10 of 64, and against 49 with thinking on: −17.2 pp [−29.7, −4.7]) and on 128 additional IFEval prompts (83 against 68 and 68 strict passes; −11.7 pp [−21.9, −1.6] each); the original 64 were inconclusive (§6.6, Table 5, pp. 7–8; Appendix K, pp. 26–28).

W6, Q3. References, ACT, RLCF. Yes, a compilation error: the submitted PDF was built without its bibliography. The revised PDF restores all 56 references (pp. 11–15; ACT and RLCF on p. 14). ACT uses verified constraint preferences for adaptation, RLCF checklist rewards for reinforcement learning; IDOL updates no parameters and scores one fixed candidate pool with and without the added checks (§1–2, p. 2). We ran no experiment against either and claim the attribution design, not a benefit of the added constraints (§9, p. 9).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Plurality across six GSM8K settings

We followed up on the reviewers’ question of how much the selector adds when the candidate answers are held fixed. We reanalyzed all six GSM8K settings with thinking enabled in Table 17 (p. 27). These combine two sketch writers (0.6B and 7B) with candidate budgets of 4, 8 and 16. In each setting, IDOL and numeric plurality chose from the same eligible answers to the same 64 questions.

Plurality picked the correct answer more often in every setting, with gains of two to six questions out of 64. Across the six settings, weighted equally, the mean accuracy gain was 6.51 percentage points (95% paired bootstrap interval +2.34 to +11.20). We resampled questions, keeping both selectors’ results across all six settings together for each question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.