acceptodds
Under review as a conference paper at ICLR 2027

The Steering Budget: Examples beat Knobs

Abstract

Generative models are steered by turning knobs — prompts, guidance scales — but every knob saturates: push it far enough and it stops moving the property you care about. This ceiling is a budget fixed by the training data before the model exists — no knob can push past it. We introduce showing, which reaches any value the whole budget allows, not just what a knob can reach, by partitioning outputs into easy-to-compute bins, auditing each, and combining bins in the proportions needed. This audit makes the split exact: a target property's variance divides into a within-bin part (telling's reach) and a between-bin part (showing's reach), both computable from data alone, before a model exists. A second, model-facing diagnostic then confirms this split carries over to a trained model: on crystals, it correctly forecasts the carry-over order of three properties before the model was ever audited. Across two unrelated domains — image and crystal-structure generation — showing out-reaches the strongest knob baselines we could build by 4.8-26x on crystal properties and 3x on image targets, with no fine-tuning: only a different choice of examples. The same two numbers also say, in advance, when telling remains the better choice. Given showing wins, a property's curvature determines the shape of the fix: concentrate on one bin for an average goal, spread across bins for coverage. Showing further brings expressiveness a knob cannot: because the specification lives in the examples, it can steer toward a target recognized but never named.

Primary area:
generative models
Keywords:
steerable generative models, controllable generation, exemplar-based conditioning, in-context learning, diffusion models, classifier-free guidance, crystal structure generation, mode collapse and diversity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Au7VOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper considers how to achieve a desired output for a generative model by steering it, and it contrasts approaches based on "telling" (adjusting parameters) versus "showing" (providing examples). It gives analytical bounds for these two approaches, and it applies the approaches to images and crystals.

Critical strengths

The task of steering a generative model toward desired outputs is an important issue. The paper provides a formalism that might be promising for this task.

Critical weaknesses

The paper appears to have been compressed to fit the page limit to an extent that details needed for evaluating it have been completely left out. In particular, despite the repeated invocation of crystals, the actual task involving crystals is never precisely stated, making it impossible to decide whether the method is producing novel gains. Similarly, the concept of a "bin" is defined too tersely and cryptically to make it possible to follow the analysis. It might be possible to explain this lack of specifics from the first 9 pages, but there was no upper limt on the number of pages of appendix, and therefore no reason why the appendix is terse and cryptic as well.

The paper omits a lot of closely related work, including

Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392–18402, 2023.

Deckers, N., Peters, J., and Potthast, M. Manipulating embeddings of stable diffusion prompts. arXiv preprint arXiv:2308.12059, 2023.

Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-Or, D. Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.

onen, K., Jentzsch, S., Diallo, D., Schütt, P., Bensch, O., Baff, R. E., Opitz, D., and Hecking, T. Style vectors for steering generative large language model. arXiv preprint arXiv:2402.01618, 2024.

Liu, S., Wang, C., Lu, J., Nie, W., Wang, H., Li, Z., Zhou, B., and Tang, J. Unsupervised discovery of steerable factors when graph deep generative models are entangled. Transactions on Machine Learning Research, 2024

Vafa, K., et al, "What's producible may not be reachable: Measuring the steerability of generative models." Advances in Neural Information Processing Systems 38 (2026): 56383-56420.

Some of these paper explicitly define the task of "steering" and also contrast adjusting instructions versus providing examples.

Questions for authors

Can you provide the full details on how you're defining "bins" and using them in the analysis?

Can you provide the full details on the task involving crystals, and how it relates to prior results on this task?

How does your paper relate to the work mentioned earlier in this review, and in what ways does it go beyond it?

3 comments
AuthorsOpenReview ↗

Summary: We defined "bin" concretely at its first use (Introduction and §3), substantially expanded the crystal task/model appendix with the structure-quality-vs-steerability distinction and two new benchmark comparisons against published crystal generators, and added six related-work citations with specific positioning — including a full differentiation from the closest prior work (Vafa et al., 2025) in a new appendix.

Response to Reviewer Au7V

We thank the reviewer for a close reading and for the related-work pointers — several of these (especially Vafa et al., 2025) are genuinely useful context we had missed, and the review has improved the paper. We have revised accordingly; all changes are in the updated PDF.

  1. Defining "bins." We agree the term was under-defined. The Introduction now defines a bin inline at first use ("picks one bin to generate into: a group of outputs that all share that value"), and §3's formal treatment now grounds the definition concretely where the notation is introduced ("all outputs sharing one label, e.g. every image π calls 'golden retriever,' or every crystal built from the same element set") rather than relying on the notation table alone, which previously just restated the symbol.

  2. The crystal task. We expanded Appendix D into "The crystal task and model," which now states the task precisely: the conditioning format, the generation order, and — most importantly — the distinction between structure quality (SUN/MSUN: is the output viable materials chemistry at all) and steerability (did the output have the property that was requested), which are different axes that can diverge. We also added a "Prior results, in context" subsection with two benchmark comparisons against published crystal generators (CrystaLLM, OMatG, ADiT, MatterGen, CDVAE), showing our generator is competitive, not a deliberately weak baseline, and that exemplar choice alone moves output quality by about as much as switching to a different published architecture does — the basis for judging whether §4's crystal results reflect a real effect rather than an artifact of a weak generator.

  3. Related work. We read all six papers and added them with specific positioning, not just citations: four (InstructPix2Pix, Prompt-to-Prompt, Deckers et al., GraphCG) are further telling/knob examples now in §2's Controllable-generation paragraph; style vectors (Konen et al., 2024) are discussed as an interesting boundary case — derived from exemplars but still a single per-output push, hence telling by our definition (Appendix H). Vafa et al. (2025) is the closest prior work: they find the same telling/showing asymmetry we predict, but empirically, by user study. §2 now states this, and Appendix H gives the full differentiation — our contribution is a data-side quantity, computable before a model or user study exists, that predicts the size and direction of that gap in advance, including when telling wins instead.

We hope these changes address the reviewer's concerns, and we're happy to expand further on any of the three points.

Thank you for these clarifications and additions. I have raised my score.

Thank You

Reviewer_H4tMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper discusses two categories of approaches used for steering generative models towards a target property: telling and showing. Telling methods steer by turning knobs (like prompts or guidance scales, e.g. classifier free guidance/CFG) while showing methods steer based on a given example set. The main point of the paper is that the reach of the two methods are very distinct and that showing’s reach towards achieving target property is higher. It shows that variance of a target property divides into within-bin and between-bin parts: telling is restricted to within-bin part (so it has limits to how much it can change target property until it stops), while showing can reach between-bin parts, making its reach higher. It then shows that this bound on reach is fixed by training data itself, before the model exists, and carries over to the model as well. It justifies showing’s higher reach over knobs with evaluations on image and crystal-structure generation, considering multiple targets.

Critical strengths

The authors list four sub-claims around the main argument that telling and showing have different reaches towards target property, and that showing beats telling. They then justify their claims considering two different datasets. Overall, their experimental settings and results support the four claims made.

Critical weaknesses
  1. I have a concern regarding their main argument that “telling can only reweight the outputs within bin and never move weight between them” (as mentioned in their Introduction). I think this is a strong claim. Considering CFG: the claim seems reasonable when CFG is guiding for the same condition based on which the data binning is done with (e.g. digit) or when target property is uncorrelated with the binning property. However, in real-world datasets, correlation between properties is not unusual. For instance, in CelebA (where, for instance, gender and hair color are correlated), binning based on gender but steering for hair color as target property may change the probability distribution (and w_b) of bins due to the correlation. In that case, doesn’t it mean that the knob (telling) is reaching beyond within-bin? Discussing (with experiment if possible) showing whether or how the claim applies to such cases will be useful.

  2. Although the paper's overall argument is understandable, the writing is difficult to follow and unpolished in several places (e.g. Introduction, Sections 3.2 and 3.3). For example: a. The term “telling” is brought up in introduction without defining first, so it reads confusing. The definition first occurs in Section 3.2: naming the goal (telling) instead. b. Because the term reweight also arises abruptly in introduction, its purpose becomes unclear. It is only defined later in Section 3.1 (in The two reaches section): “Telling and showing are two ways to reweight the same distribution -- to change which outputs the batch draws more of”. c. Including references for the mathematical claims in Section 3.2, e.g. for “if F is strictly concave its maximum is interior” will be useful. d. The term “modest sample” is used in multiple places; it will be better to define what does modest sample mean.

Questions for authors

Please refer to weakness point 1.

5 comments
AuthorsOpenReview ↗

Summary: We scoped the "telling cannot reach the between-bin budget" claim to flag the case where a knob is correlated with the key — stated explicitly in §3.1, with a new Appendix E discussion and worked example — and fixed four specific writing/definition issues (naming "telling" in the Abstract, glossing "reweight" at first use, citing a reference for the concavity claim, and anchoring "modest sample" to where it's justified).

Response to Reviewer H4tM

We thank the reviewer for a careful read, and especially for Weakness 1 — a genuinely sharpening question about the scope of our structural claim that we had not stated precisely enough. We have revised accordingly; all changes are in the updated PDF.

  1. Correlated keys and knobs (Weakness 1, Questions). The reviewer is right that the data-side split itself (T + E = Var(L)) is unaffected by correlation — it holds for any (π, L) by construction, including CelebA binned by gender with hair color as the target. What the reviewer's example tests is a different claim: whether a specific knob, in a specific trained model, actually stays inside T. We had stated the "telling cannot reach E" claim without flagging that it assumes the knob leaves the bin marginal w_b alone — true when the key and the knob are independent conditioning channels, but not guaranteed when they are correlated in the data and the model has learned that correlation (exactly the gender/hair-color case). We now say this explicitly in §3.1 and give the CelebA example and its resolution in a new Appendix E paragraph ("Correlated keys and knobs"): this is a carry-over failure on telling's side, of the same kind §3 already tests on showing's side (stability not carrying, §4.1), and is checkable with the same δ, ρ diagnostics, applied to w_b rather than g_b, v_b. We have not run that check as a dedicated experiment, but our existing Claim 2 result (§4.1) is suggestive: across every CFG level tested (1–15) and every image target, including ones plausibly correlated with class such as "animal," the realized shift stayed under telling's within-bin ceiling — leakage into E would tend to push realized shifts above it. We say this plainly in the Appendix rather than overclaim it as a dedicated test; a correlated-attribute experiment built to test this directly (CelebA or similar) is good future work.

  2. Writing (Weakness 2). All four points are fixed directly in the prose, not just noted: a. The Abstract named "showing" explicitly but never did the same for "telling" — it now does, in the same sentence that introduces knobs. b. "Reweight" appeared in the Introduction a full page before being glossed — it is now glossed inline, at its first use in the brightness-knob example. c. The concavity claim in §3.2 now cites Boyd & Vandenberghe (2004). d. "Modest scored sample," used several times without being pinned down, is now anchored at its first formal use in §3 ("few enough to stay cheap, many enough to estimate reliably"), pointing to Appendix F's granularity tradeoff where that claim is actually argued.

We hope these changes address the reviewer's concerns, and we're happy to expand further on any of the points above.

Thank you for the rebuttal and the additional discussion on correlated keys and knobs. This helps clarify a potential limitation of telling that showing can sidestep.

One minor terminology-related confusion: In Section 3 and Claim 1 of Section 4.1, the "carry over assumption" concerns whether the trained model reproduces the data's per-bin target means and variances (g_b and v_b respectively). However, the discussion in Appendix describes changes in bin proportions (w_b) under guidance as carry over failure. Aren't these distinct properties? Can't a model reproduce the data's per-bin statistics while a guidance knob still change the realized bin proportions due to learned attribute correlations? Could the authors please clarify this?

AuthorsOpenReview ↗

Summary: We confirmed that the carry-over (g_b, v_b fidelity) and the Appendix E w_b-drift discussion are distinct assumptions – a model can pass one and fail the other. Named the latter "knob independence," separated it from carry-over at both its mentions (§3.1, Appendix E), and removed the incorrect "carry-over failure" label that had cited §3's Assumption box.

Thank you — this is a real conflation, not a misunderstanding on your part, and you're right to press on it.

These are two different assumptions. §3's formal "Assumption (carry-over)" is specifically about g_b and v_b: does the trained model's per-bin mean and variance match the data's, at a given, fixed w_b. It says nothing about w_b itself. The round-1 addition to Appendix E ("Correlated keys and knobs") then called a knob shifting w_b "a carry-over failure," citing that same Assumption box — but that box doesn't cover w_b, so the label was wrong. Yes: a model can reproduce the data's per-bin g_b and v_b exactly, bin by bin, and a knob can still move w_b itself if it operates through a channel the model has learned is correlated with pi. The two can hold or fail independently.

§3.1 already stated the real prerequisite correctly, just without naming it: telling's "cannot reach E at all" claim holds "so long as the knob itself leaves pi's bins alone." That condition is not carry-over; it's a separate structural assumption about the knob's relationship to the key, which we're now calling knob independence. We've revised the paper to (1) add a closing sentence to the carry-over Assumption box itself, at its first appearance in §3, flagging that this separate assumption exists and pointing to where it's treated, so a reader meeting carry-over for the first time already knows there are two distinct things rather than finding that out later, (2) name it at its informal first use in §3.1, and (3) rewrite the Appendix E paragraph to state the distinction directly and give knob independence its own formal diagnostic, chi^2_pi(knob) = sum_b (w_b(G,knob) - w_b)^2 / w_b -- the same chi^2 form §3.1 already uses for effort, just applied to the realized bin marginal instead of a recipe, zero exactly when the knob leaves w_b alone. Defined, not run; passing carry-over implies nothing about it either way.

Nothing else in that paragraph changes: we still have not run the direct test (a correlated-attribute experiment is on our future-work list), and the suggestive-but-not-dedicated evidence from Claim 2 (no CFG level or image target pushed telling's realized shift past its within-bin ceiling) stands as before.

Thank you much for pushing on the terminology here — it made the paper more precise about exactly which assumption is doing the work where.

Reviewer_PZbdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper splits the variance of a target property of a generator's outputs (e.g., aesthetic score, band gap) into a within-bin term TT and a between-bin term EE, using the law of total variance over bins defined by a cheap key (e.g., ImageNet class, crystal chemistry). It argues that steering knobs such as prompts or classifier-free guidance only reweight outputs within a bin and are hence bounded by TT, whereas choosing the mix of bins to sample from ("showing") is bounded by EE, with both bounds following from Cauchy-Schwarz. The share E/(E+T)E/(E+T), computed from training data alone, is proposed as a predictor of which of the two moves a target further, and a goal's curvature as a predictor of whether to concentrate on one bin or spread across several. Experiments on DiT-XL/2-256 and a text-based crystal generator report gains of 4.8–26x on crystal properties and ~3x on image targets over the strongest knob baselines.

Critical strengths

S1. Simple and cheap. E/(E+T)E/(E+T) requires only per-bin shares, means, and variances, and can be computed before training (or from model samples, App. G). It offers a practical check for whether to steer through the conditioning distribution or within it.

S2. Two unrelated domains with uncertainty estimates. The experiments span image and crystal generation, report 95% CIs, and include a pre-registered carry-over prediction (Claim 1).

S3. Failure cases are reported. The paper flags where its assumptions break, namely stability carry-over (Claim 1), the off-distribution fine-tune (Claim 4), and the wrong-support audit (App. G).

Critical weaknesses

W1. The title does not match what is tested. No examples are shown to the model. In both domains, "showing" is realized as choosing the conditioning distribution over class labels or composition prompts (lines 113–115), i.e., stratified sampling over a categorical condition. No exemplar-conditioned method (e.g., IP-Adapter, in-context examples) is evaluated. I recommend either adding such baselines or reframing the paper around conditioning-distribution reweighting.

W2. Knobs are defined so that they cannot win. A knob is defined as an intervention that cannot change the bin, so the claim that knobs cannot reach EE holds by definition. Further, when the learned soft prompt beats bin selection on aesthetic and brightness, it is reclassified as no longer a knob (lines 429–431), which makes the central comparison difficult to falsify.

W3. The prediction rule does not hold as stated. Lines 213–214 state that the knob wins when E<TE < T. Yet file size (0.30) and aesthetic (0.27) both have E<TE < T, and bins win 6.0x and 4.6x (p. 7). Further, E/T\sqrt{E/T} is a ratio of two upper bounds and hence not the "floor" claimed in lines 210–212. Fig. 3 shows realized shifts at roughly a sixth of the ceiling.

W4. Headline gains compare unequal effort. Recipes realize χ2\chi^2 of order 100 against ~1 for knobs, and the ~800x band-gap ratio has a denominator whose CI includes zero. Put differently, the margins mostly reflect how far each method departs from the data. Relatedly, the abstract's claim of a fixed ceiling that "no knob can push past" conflicts with lines 319–320, which state that T\sqrt{T} "is not a wall." I would like to see both operations compared at matched χ2\chi^2 throughout.

W5. Key quantities and baselines are underspecified. The paper does not specify (a) how χ2\chi^2 is measured for CFG outputs, (b) what the "agreement" metric in Fig. 2 is, or (c) how text prompts, Composable Diffusion, and soft prompts are applied to a class-conditional DiT. In Fig. 2, the bonding CI spans most of the axis and stability (0.49) falls outside its pre-registered band (≲0.4\lesssim 0.4), so the ordering is not confirmed. Finally, the many coined terms (telling, showing, budget, reach, effort, recipe, lift, carry-over) make the paper difficult to parse.

W6. Limited novelty. Both core results, the law of total variance and the χ2\chi^2 bound on a reweighted mean (App. A), are standard. Sec. 2 omits the closest prior work (e.g., importance weighting and distribution-shift bounds, best-of-n analyses, data-mixture reweighting).

Questions for authors

Q1 (W1). Does the budget also predict the behavior of actual exemplar conditioning (e.g., IP-Adapter) on the same targets?

Q2 (W3). How do you reconcile the file size and aesthetic results with the E<TE < T rule in lines 213–214?

Q3 (W4). In the best-of-twelve comparison (χ2≈5.3\chi^2 \approx 5.3), was the bin recipe held to the same χ2\chi^2? If not, what are the realized shifts at matched effort?

Q4 (W4, W5). How is χ2\chi^2 estimated for CFG and soft-prompt outputs, and over which discretization?

Q5 (W5). How are "ask for animals," Composable Diffusion, and the learned soft prompt implemented on the class-conditional DiT-XL/2-256?

Q6 (W5). What is the "agreement" metric in Fig. 2, and can the timestamped pre-registration be shared with the AC?

4 comments
AuthorsOpenReview ↗

Summary: Clarified "showing" means bin-mixture control, not exemplar-conditioning networks; backed the knob/soft-prompt distinction with a direct measurement; corrected an overclaim in the equal-effort comparison (√(E/T) is a ceiling, not a guaranteed floor on realized outcomes); filled in five underspecified items, including a wider-than-expected carry-over CI; and added missing citations plus a direct statement of the paper's novelty.

We thank the reviewer for a careful, technically substantive read that caught real imprecision in several of our claims. All changes are in the updated PDF.

  1. Title/mechanism scope (W1, Q1). Agreed: "showing" reads as literal exemplar-conditioning (IP-Adapter), while §4 tests choosing bin proportions directly. Fixed by stating this explicitly at first use (Intro, §2): the handoff can be literal (untested here) or direct (what we test); the budget bounds the operation, not the mechanism. Q1: we have not tested exemplar-conditioning and say so plainly; we believe the theory extends to it, but that's unverified.

  2. "Knob" definition and soft-prompt (W2). "Knob" formalizes a practice already recognized informally (prompts, guidance scales, conditioning tokens) via one property: it leaves the bin alone. That precision enabled a falsifiable test, not one chosen to win it. The soft-prompt reclassification is such a test: re-classifying its outputs with key π, independent of the scorer, shows 0% retain the conditioned class for the two targets it wins (aesthetic, brightness), vs. ≥92% for every CFG knob — now stated in §4.1.

  3. Equal-effort "floor" claim (W3, Q2) — the deepest issue, revised most. Both points correct. √(E/T) is a ratio of two Cauchy–Schwarz ceilings at matched effort; it does not guarantee which operation wins a realized (unequal-effort) experiment, since realized shifts sit below their own ceiling by varying amounts (Fig. 3). Dropped "floor"/"no knob strength can change" language beyond what the ceiling supports. The "small share ⟹ knob wins" claim was also too broad: file size (√(E/T)=0.65) and aesthetic (0.61) have shares as small as brightness's (0.45) and still lose to bins (6.0×, 4.6×). The realized ~10× effort gap (χ²≈1 for knobs vs. ≈100 for recipes) predicts aesthetic/file size well (6.1×, 6.5× vs. actual 4.6×, 6.0×) but not brightness (predicted 4.5×, actual 0.5×) — brightness's recipe realizes an unusually small fraction of its own ceiling. §3.1 and §4.1 rewritten accordingly.

  4. Unequal-effort comparisons, "not a wall" (W4, Q3). Matched-effort numbers already existed (√(E/T)); point 3 makes the reconciliation explicit. Reworded "√T is not a wall telling hits" (misreadable against the Abstract's "no knob can push past it") to "telling falls well short of that ceiling, nowhere near pushing past it." Q3: in the crystal best-of-twelve comparison, the bin recipe was not held to the knob's χ²≈5.3 — the knob's own near-ceiling; it ran at its usual χ²≈100. Reworded, with the true equal-effort numbers added: √(E/T) = 6.8×/3.2×/1.1× for gap/density/stability — showing still wins at matched effort on all three.

  5. Underspecified metrics and baselines (W5, Q4–Q6). Five fixes: a. χ² for CFG: the same histogram-divergence construction as showing's χ²(μ_P‖J_P), applied within one bin against the CFG=1 baseline (§4.1). b. Fig. 2's "agreement" is per-bin-mean r² — already named for images, now labeled for crystals too. c. DiT-XL/2-256 implementation: new Appendix I — "ask for X" = uniform sampling over X's top-300 audited classes; Composable Diffusion (Liu et al., ECCV 2022, now cited) = standard product-of-experts composition; soft-prompt = the per-block embedding from point 2. d. Bonding's carry-over CI: [0.04, 0.97] — nearly the full range, not separating it from density or stability on its own. Added to §4.1; "confirmed the ordering exactly" softened to "confirmed in point estimate... though bonding's own CI is wide." Happy to share the pre-registration commit with the AC. e. "The many coined terms... difficult to parse" — which all three reviews raise. Audited all eight terms named: six are load-bearing and kept; dropped "crossover" as a redundant second name for E/(E+T); flagged "lift" as secondary/skippable; added telling/showing/budget to the notation table (previously symbol-only), with a forward-pointer from the Intro.

  6. Related work and novelty (W6). Agreed the law of total variance and Cauchy–Schwarz bound are standard — that's not the novelty. The novelty is a predictive quantity built from them, E/(E+T), computable from data alone before any model exists, that calls in advance how large the telling/showing gap is, which operation wins, and which recipe shape closes it. Vafa et al. (2025, cited in §2/Appendix H) find the same asymmetry empirically, after the fact, by user study; we predict its size and direction in advance instead, verified by pre-registered predictions across two domains sharing no model, data, or code. Added the four missing citations (Shimodaira 2000; Cortes et al. 2010; Gao et al. 2023; Xie et al. 2023) in a new §2 paragraph ("Reweighting tools").

The "floor" claim in point 3 was a real overclaim, now corrected. Happy to expand on any point above, or share the pre-registration commit (point 5d) with the AC.

I thank the authors for their response! A quick response contrasting the points against the revised PDF:

(W2), (W5b), (W5c), (W5e), (W6): Acknowledged, and sounds good.

(W1): I don't see this in the title or Abstract, where "showing" is first used.

(W3): I don't see this in the Abstract, Contribution 2, l. 254, l. 396–397, l. 488, App. H, or the Conclusion.

(W4): I don't see realized shifts at matched χ² in §4.1 or elsewhere in the PDF.

(W5a): I don't see the histogram discretization in §4.1.

(W5d): I don't see this in the Fig. 2 caption or the Abstract.

AuthorsOpenReview ↗

Summary: Replaced the W1 language with a positive claim — exemplar handoff already works via a bin histogram (Appendix C), with "untested" now scoped to just the trained exemplar-conditioning network. Swept the E/T-predicts-the-winner overclaim (W3) to four more locations. Disclosed the crystal equal-effort number is a ceiling, not a measured run (W4), with an offer to run it if time allows. Added CFG's histogram discretization (W5a) and bonding's wide CI to the figure/Abstract (W5d).

We thank the reviewer for checking our round-1 fixes directly against the revised PDF. Four of the five items below were real misses — the fix landed in one place but not every place you correctly checked — and one (W4) turned out to be a genuine unmeasured gap, not just a missing sentence. All changes are in the updated PDF.

  1. Showing vs. literal exemplars (W1) — not just a location fix, a sharper one. You're right the Abstract still read as if "showing" meant literal exemplar-conditioning. On reflection, our round-1 fix — hedging that exemplar handoff is "a related mechanism we connect to but do not test" — undersold what the paper already does. One advantage we claim is expressiveness: a curator or materials scientist can hand over examples of a property they recognize but can't articulate. We have always had a mechanism for turning that handoff into a weighted distribution over bins — read each example's bin off π, and the histogram is the recipe (§3.3, Appendix C) — the same recipe as the direct form tested throughout, not a different one; Claim 3's lift diagnostic (rank correlation 0.94 to realized payoff) is direct evidence for it. We simply hadn't said so where "showing" is first used. We replaced the hedge with this positive statement in the Abstract and Introduction, re-scoped §2's "we do not test it" disclaimer to the one mechanism it's actually true of — a trained exemplar-conditioning network (IP-Adapter/DreamBooth-style), a different, third thing from either form — and added a worked example to Appendix C. Please let us know if now the title is justified in using the word 'examples'.

  2. The E/T-predicts-the-winner overclaim, recurring (W3). Correct — we fixed §3.1 and §4.1's main discussion in round 1 but didn't sweep the rest of the paper for the same pattern. Found and reworded it in four more places — the Abstract, Contribution 2, §4.1's Claim 2 close, and the soft-prompt result sentence — each softened from "predicts/says in advance which one wins" to language that bounds and flags the low-share cases without claiming the share alone decides them. The clearest instance was Appendix H's comparison to Vafa et al.: "called correctly, in advance, in our own experiments" became "tested directly in our own experiments, where it closes for one of three low-share targets, and where we trace the rest to the realized effort gap and the recipe's own slack rather than to share alone." We checked the Conclusion specifically: "it predicts which targets a knob cannot reach" is a structural claim (telling can never cross into E), not the realized-winner claim the other locations had wrong — left as is.

  3. Equal-effort crystal comparison (W4) — a genuine gap, not a wording miss. You're right: the √(E/T)=6.8/3.2/1.1× sentence near the crystal best-of-twelve result read as a measured matched-effort run. It is the theoretical ceiling from §3.1's Cauchy–Schwarz bound, not something we ran — constructing and measuring it needs new crystal generation at a recipe throttled to the knob's χ²≈5.3, scored through the MACE stability pipeline, compute we have not yet spent. We reworded both passages to state plainly that this is a bound, not a measured tie-breaker, and that a realized run can sit below its own ceiling. We're telling you this directly rather than leaving it implied, and will update this thread if we manage to run it before the period ends (we did; please see comment below).

  4. CFG's χ² discretization (W5a). Fair miss — the detail existed in the experiment code but never made it into the text. The pooled range of the knob's output and the CFG=1 baseline is cut into 12 equal-width histogram bins, Laplace-smoothed, with the same χ² formula applied to those bin counts instead of μ_P versus J_P. Added to §4.1 directly.

  5. Bonding's wide CI, in the figure and the Abstract (W5d). Right — the hedge we added to §4.1's body text in round 1 never reached Fig. 2's caption or the Abstract's carry-over sentence, so a reader skimming either would see an unqualified "confirmed." Added the actual 95% CIs to the caption (density [0.91,1.00], bonding [0.04,0.97], stability [0.17,0.71], pulled from the exact bootstrap arrays rather than re-derived) and a short hedge to the Abstract's carry-over sentence.

Thank you again for checking this so carefully — each of these made the paper more honest about what it has and hasn't actually shown.

AuthorsOpenReview ↗

Summary: Ran the W4 matched-effort experiment we offered to attempt. Showing beats best-of-twelve by 18.2x (gap) and 5.9x (density) at matched χ²≈5.3, exceeding the ceiling ratio √(E/T) because that ratio bounds ceilings, not realized shifts. Stability turned out infeasible to measure on the same scorer (training data saturated); a substitute proxy showed no detectable gain. Updated §4.1 with the real numbers.

Follow-up on W4: we said we'd update this thread if we managed to run the equal-effort, matched-χ² experiment before the discussion period closed — we did.

Throttled to the knob's own χ²≈5.3, showing beats best-of-twelve by 18.2× on gap and 5.9× on density — both above the ceiling ratio √(E/T)=6.8×/3.2× we quoted in the revision. That's not a contradiction: √(E/T) bounds the ratio of the two methods' ceilings, not the ratio of what each realizes, and best-of-twelve realizes well below its own ceiling here.

Stability turned up something we hadn't anticipated: the training data are too saturated (94.5% of chemistries already at the maximum metastable rate) for any recipe — showing or telling — to reach χ²≈5.3 on the same scorer at all, so the matched-effort comparison is undefined there, not merely unmeasured. A substitute recipe on a different (DFT-based) proxy showed no detectable gain, where the ceiling had suggested 1.1×.

We've updated §4.1's best-of-twelve paragraph in the PDF with the real numbers for all three properties, including this caveat. Thank you again for pushing on this — it surfaced a genuine boundary condition in the crystal data that we hadn't previously measured.

Comments on the paper

1 comment
Area_Chair_urrKOpenReview ↗

Requesting reviewer feedback

Dear Reviewers,

Thank you for your reviewing service. The authors have provided detailed rebuttals to the comments. Please confirm that you have read them, state which of your concerns are or are not addressed, and whether that changes your recommendations. Thank you Reviewer PZbd for your continued engagement; requesting others to join.

Thank you,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.