acceptodds
Under review as a conference paper at ICLR 2027

When Acceptable Answers Share Parts, Feedback Helps More

Abstract

A correct observation about one part of an answer can help with many acceptable answers, or with very few. We study whether this shared structure changes the value of component-level feedback. Our controlled task asks language models to recover five terms from a forty-term catalogue. We vary overlap among acceptable answers while holding their number, the numerical data, report format, and verifier allowance fixed. Across 24 families and 11 models, the graded-recovery advantage of the full over a profile-substituted report increases with overlap: the interaction is +0.419 [+0.323, +0.508] per unit of a chance-corrected overlap index. Correct-minus-no-report recovery changes from −0.012 at zero overlap to +0.366 at high overlap. A calibrated mechanical reader, defined by an explicit expected-overlap objective, has a graded interaction of +0.440 and a larger exact-recovery interaction than the models. Fixed-report and proposal-conditioned studies on external SAT formulas show related positive associations; the fixed-report slope is +0.169 [+0.086, +0.251] after answer-count control. The construction, controls, and reader comparison show how a task property changes the usefulness of a specified report. They also separate recovering useful parts from assembling an acceptable whole. Code and data: https://anonymous.4open.science/r/verified-feedback-discovery-ready-7381/README.md.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Large language models, Neurosymbolic AI, Verifier-guided reasoning, Feedback-based refinement, Controlled evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KY9dOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper investigates how overlap among acceptable solutions affects the usefulness of component-level feedback for LLMs. The authors construct a controlled symbolic regression task in which LLMs select five mathematical terms from a predefined grammar. They vary the overlap among 32 designated acceptable solutions while keeping the numerical data, solution count, and evaluation procedure fixed.

Experiments across 11 models show that component-level feedback becomes more effective as acceptable solutions share more components. Additional experiments using a mechanical reader and SAT problems provide supporting evidence.

Critical strengths
  1. The experimental design systematically controls several factors, including the number of acceptable solutions, numerical data, and feedback format.

  2. The evaluation includes multiple LLMs, feedback conditions, and a mechanical baseline.

  3. The authors distinguish between partial recovery of correct components and exact recovery of an acceptable solution.

Critical weaknesses
  1. Limited scientific significance of the artificially defined acceptable solutions

My primary concern is the scientific meaning of the acceptable solution sets used in the main experiments.

The authors artificially designate 32 five-term combinations as acceptable solutions, although other combinations can fit the same numerical data equally well. The paper does not provide a scientifically motivated criterion for distinguishing the designated acceptable solutions from other numerically equivalent solutions.

Consequently, the task primarily evaluates whether LLMs can recover elements of a researcher-defined subset rather than whether they can discover scientifically meaningful equations.

This distinction substantially limits the significance of the findings, particularly given the paper's motivation in scientific discovery and symbolic regression.

  1. The main finding may largely follow from the experimental construction

The component-level feedback is generated directly from the artificially defined acceptable solution sets. When these solutions share more components, the feedback provides information about components that are more frequently useful across acceptable answers.

Therefore, the observation that feedback becomes more effective as solution overlap increases seems largely predictable from the construction itself.

The similar trend observed with the mechanical reader further suggests that the result may primarily reflect the information structure of the task and feedback, rather than revealing a new property of LLM reasoning.

Although the experiments quantify this relationship, it remains unclear what fundamentally new insight is obtained beyond this intuitive expectation.

  1. Unclear practical implications and generalizability

The paper does not sufficiently demonstrate how the findings would improve feedback mechanisms in realistic scientific discovery or other practical LLM applications.

In real-world symbolic regression, acceptable equations are generally determined by criteria such as predictive accuracy, interpretability, physical consistency, or domain constraints, rather than membership in an arbitrarily selected subset of numerically equivalent expressions.

While the additional SAT experiments provide some evidence beyond the synthetic setting, they do not fully establish the relevance of the main findings to realistic scientific discovery tasks.

Questions for authors
  1. What scientifically meaningful criterion distinguishes the 32 designated acceptable solutions from other solutions that fit the numerical data equally well?

  2. Given that the feedback is constructed directly from the acceptable solution sets, to what extent is the observed relationship between solution overlap and feedback effectiveness an expected consequence of the experimental design?

  3. What new understanding of LLM reasoning or feedback utilization does this study provide beyond what can be explained by a simple mechanical reader?

  4. How could these findings inform the design of practical feedback systems when the complete set of acceptable solutions is unknown or cannot be enumerated?

2 comments
AuthorsOpenReview ↗

Response to Reviewer KY9d

Thank you for the review. We answer your four questions in order. We also revised the presentation.

Q1. What scientifically meaningful criterion distinguishes the 32 acceptable solutions?

The 32 solutions are authored. This keeps the number of answers fixed while the overlap changes. Section 7 now says this.

Simple criteria of the kind used in equation discovery give the same range of overlap. We checked this on the same catalogues, with the same number of answers (Appendix K.7).

How the 32 acceptable supports are chosenOverlap L
At random, among the supports that fit0.03
The 32 with the lowest expression complexity0.71

So a verifier that checks only the fit gives a component report almost nothing to say. A parsimony criterion gives it a lot to say. Our authored levels (0.00, 0.38, 0.78) test this range under controlled conditions.

Q2. To what extent is the result an expected consequence of the design?

The direction follows from the design. The abstract now says so. The experiment measures the size of the effect. It also shows that a simple rule accounts for the size. A mechanical reader that only fills each reported slot has gaps of −0.018, +0.157 and +0.283. The models' gaps are −0.056, +0.166 and +0.271. The two agree across levels and across families.

Q3. What new understanding does the study give beyond a simple mechanical reader?

It shows which simple rule describes the models, and where a component report stops helping.

  • Models fill slots and ignore the stated frequency. They follow a slot that says 3.1% as often as a slot that says 100% (difference +0.018 [−0.049, +0.097]).
  • The loss is between profiles and terms. When a reported profile belongs to one term, models commit the right term in 91% of cases. When two terms share the profile, they do so in 41% of cases. Completing an answer is not the problem: 87% of commitments with all four recurring terms are accepted.
  • The effect is stable. It holds for all 11 models, against a substituted report (+0.419) and against no report (+0.484 [+0.389, +0.582]).

Q4. How can this inform practical systems when the acceptable set cannot be enumerated?

Overlap can be estimated from a few known answers (Section 6.3). On 1,000 SAT formulas, we computed the index from five sampled solutions. It correlates 0.90 with the exact overlap, and it gives the same slope (+0.215 against +0.213). When we rank held-out formulas by this estimate, the top half gains +0.221 from the report and the bottom half gains +0.119.

Section 7 gives three design points:

  1. Estimate overlap from a few known good answers before building a component report.
  2. List only the components you want the model to use. Models follow low-frequency slots as readily as high-frequency slots.
  3. At high overlap, make the report distinguish similar alternatives.

W3. Relevance beyond the synthetic task.

The external evidence uses SAT formulas from SATLIB. It now starts with a study of feedback on models' own answers (Section 6.1). The slope is +0.620 [+0.434, +0.806] against a sham report, and +0.609 [+0.417, +0.815] against no report.

Presentation.

  • A new Figure 2 shows two reports exactly as the models saw them.
  • Table 1 now gives the plain names of the arms.
  • Section 5 is organised around three questions.
  • Section 7 starts with the contribution in one sentence.

Thank you again. We hope the revised paper is easier to follow.

AuthorsOpenReview ↗

Follow-up to Reviewer KY9d: Overlap on Published Benchmarks

Thank you again for your review. We add new evidence on your main concern: the scientific meaning of the acceptable sets (W1, Q1) and the relevance to real discovery tasks (W3). It is in an updated revision (Section 7 and Appendix K.9).

Overlap on published benchmarks. We measured the overlap index on published benchmark problems. There, the benchmark itself defines which answers are acceptable. No model is involved.

BenchmarkAcceptable answersProblems measuredOverlap L
AI-Feynman equationsSupports that fit held-out data15 of 381.00 for all
Nguyen equationsSupports that fit held-out data2 of 100.05 and 1.00
Published dynamical systems (dysts)Supports that fit held-out data102 of 2501.00 for all
Reference causal networks (bnlearn)Graphs in the Markov equivalence class23 of 240.59 to 1.00; 20 are at 0.9 or above
SAT formulas (SATLIB uf20-91)Satisfying assignments1,000 of 1,0000.18 to 1.00

This gives three answers to your questions.

  1. Scientific criteria give high overlap. On the measured AI-Feynman and dynamical-system problems, exactly one support is acceptable (L = 1). Our controlled study includes this level. There the full report gains +0.507 over no report, the largest gain in the study (Table 11).
  2. Why the sets are authored. Published scientific benchmarks do not vary in overlap. A study of how overlap changes the value of feedback therefore needs sets with controlled overlap. SAT formulas are the one external family we found with a wide range, and the relation holds there (Section 6).
  3. What the levels mean. Fit alone gives L = 0.03. A parsimony criterion gives 0.71. One true answer gives 1. Our levels (0.00, 0.38, 0.78, and 1.00) cover this range.

Your Q2, stated precisely. We added a test that separates overlap from what the report says. We regress the models' gap on a slot-filling reader's gap and on L. The coefficient on L is +0.010 [−0.122, +0.138], and the coefficient on the reader's gap is +0.88 [+0.66, +1.11] (Section 5, Appendix K.4). So the effect of overlap passes through what the report says, and the models realise most of it.

A shared concern. Your concern about evidence beyond the synthetic task (W3) is close to Reviewer xXyo's second weakness. Our reply to Reviewer xXyo gives the per-model results of the SAT study (Table 6). Reviewer xXyo has since replied and raised their rating, so that thread may also be useful.

We hope this answers your main concern. If any point remains open, we will be glad to answer it before the discussion closes. If the revision resolves your concerns, we would be grateful if you would consider updating your rating.

Thank you for your time and consideration.

Reviewer_xXyoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether feedback becomes more useful when acceptable answers share components. A controlled term-selection task varies answer overlap while holding numerical observations and the number of acceptable answers fixed. Across 24 task families and 11 models, component-frequency feedback provides larger graded-score gains at higher overlap. Mechanical-reader comparisons and SAT experiments examine the gap between recovering useful components and producing complete acceptable answers.

Critical strengths
  • Holding numerical data and acceptable-answer count fixed helps isolate the role of shared components; the feedback effect has a consistent direction across all 11 models.

  • No-report controls and separate graded and exact outcomes make the benefits and limits of feedback easier to interpret.

Critical weaknesses
  • The mechanical reader receives the full candidate catalogue and searches it exhaustively, while LLMs receive only the grammar. The exact-answer gap therefore does not isolate reasoning or assembly ability.

  • Generalization rests on observational SAT checks. The fixed-report analysis retains only one model and finds no clear exact-recovery trend; the broader own-proposal analysis is exploratory.

Questions for authors
  • As a matched-information control, how much of the exact-answer gap remains when LLMs also receive the full 40-term catalogue?

  • As a further test, would harder SAT instances reproduce the trend across models excluded by the ceiling rule?

3 comments
AuthorsOpenReview ↗

Response to Reviewer xXyo

Thank you for the review and for the two clear questions. We answer them in order.

W1 and Q1. The reader sees the catalogue and the models do not. How much of the exact gap remains when LLMs also receive the catalogue?

We agree that the gap against the catalogue reader does not isolate reasoning or assembly. The revised Section 5 no longer reads it that way.

We could not run the catalogue-in-prompt arm in the response period. We answered the question from the reader side: we removed the catalogue from the reader (Table 3).

ReaderModel − reader, exact
Catalogue reader−0.334 [−0.461, −0.222]
Same reader, without the catalogue−0.126 [−0.211, −0.038]
Slot reader−0.101 [−0.192, 0.000]

About three fifths of the gap is catalogue access.

The models' own answers show where the rest is. When a reported profile belongs to one term, models commit the right term in 91% of cases. When two terms share the profile, they do so in 41% of cases (Section 5, Appendix K.5). This is the step at which the catalogue helps the reader.

W2 and Q2. Would harder SAT instances reproduce the trend across the models excluded by the ceiling rule?

Yes. Section 6.1 now starts with a prospectively specified study in which GPT-5.5 and GPT-5.6, the two excluded models, work on the harder uf50-218 formulas. In this study the report is feedback on the model's own first answer. The outcome is exact solving. Slopes are clustered by formula (Table 6).

ModelFormulasFull − shamFull − none
GPT-5.5uf50-218+0.585 [+0.215, +0.979]+0.685 [+0.306, +1.081]
GPT-5.6uf50-218+1.414 [+0.881, +1.916]+0.744 [+0.230, +1.261]
Opus-4.7uf20-91+1.297 [+0.828, +1.788]+0.716 [+0.240, +1.192]
Opus-4.8uf20-91+0.436 [+0.041, +0.837]+0.494 [+0.115, +0.917]
Sonnet-4.6uf20-91−0.018 [−0.418, +0.375]−0.080 [−0.525, +0.330]
  • The two GPT models together give +0.999 [+0.694, +1.303]. With a control for the number of solutions, the slope is +0.890 [+0.558, +1.212].
  • All five models together give +0.620 [+0.434, +0.806] against the sham, and +0.609 [+0.417, +0.815] against no report.

On the fixed reports, GPT-5.6 also has a positive exact slope on uf20-91: +0.612 [+0.458, +0.768] (Section 6.2).

Thank you again for the suggestions.

Reviewer_xXyoOpenReview ↗

Response by Reviewer

Thank you for addressing my concerns. I will raise my original score.

Thank you for reading our response and the revised paper, and for raising your score. We are glad that the changes addressed your concerns. If anything else would be useful to clarify before the discussion closes, we are happy to do so.

Reviewer_XBFnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the value of component-level feedback to an LLM depends on how much the acceptable answers to a task share components.

The main study is a controlled term-selection task. Each of 24 generated families has a 40-term log-monomial catalogue, and many 5-term supports fit its data exactly. Within each family, a declared set of 32 acceptable supports is built at three levels: balanced, two-term core and four-term core. Overlap is summarized by a chance-corrected index L (Eq. 1), with values of about 0, 0.38 and 0.78. The data, catalogue, answer count, support size and report schema are held fixed across levels.

Models receive a fixed report: a fit score plus five profile/frequency slots for the most frequent components. This full report is compared with a partial sham (three of five profiles replaced), an all-swapped sham, a scalar-only report and no report (Table 1). Outcomes are graded recovery G (best F1 against an accepted support) and exact recovery E (Eq. 2).

Across 11 LLM endpoints, the full-minus-partial-sham graded gap increases with L. The family-clustered slope is +0.419 [+0.323, +0.508] (Sec. 4), and full-minus-none goes from -0.012 to +0.366 (Table 2). A calibrated "expected-overlap reader" enumerates the catalogue and maximizes Eq. 3. Its graded interaction is similar to the models' (+0.440), but its exact-recovery interaction is much larger (+0.486 vs +0.151; difference -0.334 [-0.461, -0.222], Table 3).

Two SATLIB studies report positive associations with observed L. The first is a fixed-marginal-report study on uf20-91, with one model retained after a ceiling rule (Sec. 6.1, App. F). The second is an exploratory own-proposal revision study on 90 formulas with 10 models (Sec. 6.2, App. G). The appendices add a reader ladder (App. H), a 20%-complete Boolean-circuit prefix (App. I.1), a null pair-credit probe (App. I.2) and replay documentation (App. J).

Critical strengths
  1. Careful, transparent design and inference. The design holds the data, catalogue, answer count, support size and report schema fixed while changing the accepted set (Sec. 3.1). Inference uses the 24 families as units, not the 3,168 cells (Sec. 3.3, App. B.5). Absolute arm means are reported next to the contrasts, including the falling no-report baseline (Tables 2, 4, 10). Nulls are reported (App. I.2; Sec. 6.1). This matters because the numbers can be trusted as measurements of what was run. My concerns are about what they mean.

  2. The explicit-reader comparison prevents over-attributing the graded result to LLM reasoning. A transparent marginal-trusting rule reproduces both the models' graded interaction (+0.440 vs +0.419; difference -0.021 [-0.102, +0.058], Table 3) and the falling no-report baseline (0.384 to 0.245, Sec. 5). The paper states this directly: "The shared trend is not specific to a language-model consumer" (Sec. 5). This is the correct way to read the main result, and it is an evaluation practice others could adopt. One qualification, from Fig. 2: the matching slopes hide a roughly constant shortfall. In point estimates, the models' gap is below the reader's at every level (by 0.056, 0.058 and 0.073). The slopes agree because these shortfalls are similar. Only at L=0 are the two separated pointwise. At L=0.38 and L=0.78 the pointwise 95% intervals overlap: the model upper bounds are 0.206 and 0.335, and the reader lower bounds are 0.185 and 0.306.

Critical weaknesses
  1. The report's decision value rises with L by construction, so the design cannot separate an effect of overlap on feedback value from this report simply being more informative at high L.

    The full report lists the top-5 marginals of S_f, and its content changes with the level:

    • At L=0, every component is in 4/32 supports. All five slots read 12.5%, and the five profiles are chosen by catalogue-index tie-breaking (App. A.3, A.5). The report therefore cannot indicate which components to choose.
    • At L=0.78, four slots read 100% and point to profile classes containing the four core terms. Profiles are not term identities (Sec. 5, p. 6), but a commitment containing the core has G ≥ 0.8.

    The authors acknowledge this coupling: "The report construction deliberately makes recurring components easier to identify" (p. 2), and "The matched format does not equalize decision value across overlap levels" (Sec. 3.2). The encoding-size check is explicitly not an information measure (App. A.5).

    The scalar and in-loop fit-score channels carry data-consistency information but no acceptance information. Every rank-four support fits exactly, and the fit-score field "is one in this construction" (App. A.2, A.5). So scalar ≈ none is expected. Sec. 4's statement that the controls "locate the useful signal in the component report rather than the mere presence of a fit-score field" adds little, and the "feedback" is effectively a fixed hint repeated each round (App. B.1).

    Three further points reduce what the slope shows:

    • The slope is driven by the endpoints. Family L values are about 0, 0.382 (mean across families) and 0.779 (App. A.3), so the middle level gets almost no OLS weight. From Table 2, (0.631-0.360) - (0.379-0.435) = 0.327, and 0.327/0.779 ≈ 0.42, which reproduces +0.419. The level gaps (-0.056, +0.166, +0.271) are monotone, so the ordering is descriptive support, but "per unit of L" (Sec. 3.3, p. 4) is not a tested dose-response. Fig. 2 also shows the response is concave. The model gap rises by 0.222 from L=0 to 0.38, but by only 0.105 from 0.38 to 0.78 (about 0.58 vs 0.26 per unit L). The pointwise 95% intervals at the two upper levels just touch: [+0.125, +0.206] and [+0.207, +0.335], read from the figure's vector data. The reader has the same shape (+0.224, then +0.120), so the shape, like the slope, is reproduced by a simple rule.
    • The primary contrast changes meaning across levels. At L=0, all reports are frequency-correct, yet the partial sham beats the full report by 0.056 (App. C.1). Fig. 2 shows that this is not noise: its pointwise interval, [-0.090, -0.022], excludes zero, whereas the reader's gap is 0.000. This model-specific negative intercept supplies about 17% of the +0.419 slope (0.056/0.327). At higher L, the sham attaches 100% frequencies to decoy profiles (Sec. 4), so part of the contrast measures harm from misinformation. Full-minus-none avoids this, but it is reported only descriptively (Table 2 caption).
    • The pair-credit variant cannot test generality. It is the only variant that changes the report type (App. I.2), and it uses three models over a pair-overlap range of only 0.031-0.211, with an interval of [-0.40, +0.44].
  2. The conclusion that the study can "separate recovering useful parts from assembling an acceptable whole" (abstract; see also Sec. 5, p. 7; Sec. 7) rests on a reader with privileged catalogue access, and the paper's own appendix shows the exact gap is reader-dependent.

    The paper interprets the exact model-minus-reader gap (-0.334, Table 3) as models turning part-level information into accepted wholes less well. Several facts undercut this reading:

    • Information access differs. The reader enumerates all 658,008 supports of the explicit 40-term catalogue and reads unrounded frequencies (Sec. 5; App. D.1, D.3). The models see only the grammar (App. B.1). At L=0.78, a model must generate a fifth term that happens to be in the catalogue and in S_f, whereas for the reader "any of 32 of the other 36 catalogue terms would complete an accepted support" (p. 6). The paper calls these "additional resources" (Sec. 5) but still draws the parts/whole conclusion.
    • The gap reduces to one cell. No model commitment is accepted in any non-full arm (0/6,336 across four arms, p. 7), so the models' exact slope is about 1.29 × their full-report exact rate at L=0.78. That rate is 62/528 = 0.117, against the reader's 0.375. The reader's sham exact rate is not reported.
    • The model-side exact result rests on few events. It is based on 64 accepted commitments in total (Table 4), with an unexplained even/odd family split of 1.89% vs 6.69% (App. C.2).
    • A different reader removes the gap. The paper notes general reader dependence (Sec. 5; App. E.1), but leaves a decisive number in App. E.1: against the calibrated true-set selector, the exact model-minus-reader difference is -0.001 [-0.080, +0.088] (p. 22). That selector is informationally privileged (it receives the true S_f), and its graded interaction is weak (+0.221, Table 12), so exact parity may reflect its limited candidate generation. Even so, the -0.334 gap belongs to the exhaustive-enumeration reader, not to an established LLM-specific assembly failure.
Questions for authors

see above

3 comments
AuthorsOpenReview ↗

Response to Reviewer XBFn (Part 1/2): Weakness 1

Thank you for the detailed and careful review. We agree with both main weaknesses, and we have revised the paper accordingly. We reply in two parts.

W1. The report's decision value rises with L by construction.

We agree. The abstract and Section 1 now state that the direction of the effect follows from the report's design. The revised Section 5 measures what the design does not fix: how models use the report, and how much of its worth they realise.

  • Models fill the reported slots. Under the full report, models follow 0.80, 0.85 and 0.89 of the reported slots at the three levels. They do not weight slots by stated frequency. At L = 0.78, the 100% slots and the 3.1% slot differ by +0.018 [−0.049, +0.097].
  • A slot-filling reader tracks the models. This reader fills each reported slot with a random term of the reported profile. It uses no catalogue. Its gaps are −0.018, +0.157 and +0.283. The models' gaps are −0.056, +0.166 and +0.271. The agreement also holds within levels, across families: r = 0.69 [0.55, 0.80].

The constant shortfall (your note on Strength 2). This shortfall is relative to the catalogue reader. Against the slot reader, the difference in interaction is +0.033 [−0.029, +0.091]. Without the 0.833 failure factor that all readers share, the models are 0.02 to 0.07 below the slot reader at each level (Appendix K.4).

Scalar ≈ none, and the "fixed hint". We agree. Section 3.2 now calls the scalar arm a negative control. We removed the sentence about locating the useful signal. Feedback on a model's own answer is now the first external study (Section 6.1).

The slope is driven by the endpoints. We agree. Section 4 now tests each step. The gap rises by +0.222 [+0.174, +0.269] from L = 0 to 0.38, and by +0.105 [+0.034, +0.178] from 0.38 to 0.78. It is monotone in 19 of 24 families. The middle level is +0.061 [+0.011, +0.110] above a straight line, so the response is concave, as you noted.

The primary contrast changes meaning across levels. Full-minus-none now has a full test: +0.484 [+0.389, +0.582], positive in all 24 families. The partial sham is above no report at every level (+0.044, +0.096, +0.095). So the widening gap comes from the full report gaining.

The reversal at L = 0. You are right that it is not noise. Two properties of the display contribute (Section 4, Appendix K.6):

  • At L = 0 the full report shows three distinct profiles, and the partial sham shows 4.8 on average. The slot reader scores 0.022 [−0.007, 0.046] lower with the full report for this reason.
  • Models more often commit a repeated term under the full report (37 commitments against 7). These score zero. If we score them by set F1, the gap is −0.039 [−0.070, −0.009], and the two interactions are +0.413 and +0.479.

Pair credit. We agree. We draw no claim of generality from it (Section 7).

AuthorsOpenReview ↗

Response to Reviewer XBFn (Part 2/2): Weakness 2

W2. The parts/whole conclusion rests on a reader with catalogue access.

We agree. We removed this conclusion from the abstract. Section 5 now shows how the exact difference depends on what the reader knows (Table 3).

ReaderModel − reader, exact
Catalogue reader (as submitted)−0.334 [−0.461, −0.222]
Same reader, without the catalogue−0.126 [−0.211, −0.038]
Slot reader−0.101 [−0.192, 0.000]
True-set selector (Appendix E.1)−0.001 [−0.080, +0.088]

About three fifths of the submitted gap is catalogue access.

The gap reduces to one cell. Yes. This also holds for the reader. Its exact rates with the partial sham are 0.001 at L = 0 and 0.000 at the other two levels (Table 4). We therefore analyse this cell directly, without a reader. Of the 528 full-report commitments at L = 0.78:

  • 365 (69.1%) contain all four recurring profiles;
  • 71 (13.4%) contain the four recurring terms;
  • 62 are accepted.

So models do not lose exact recovery when they complete an answer. They lose it between profiles and terms.

Where the loss is. We split the 96 recurring slots by whether the reported profile belongs to one term or to two (Section 5, Appendix K.5).

Reported profileSlotsRecurring term committed
Belongs to one term2991% [88, 95]
Shared by two terms6741% [32, 51]

The exact rate of a family falls with its number of shared slots: 0.21, 0.07 and 0.00 for two, three and four shared slots.

Few events, and the even/odd family split. Five families hold 50 of the 62 accepted commitments. The split follows which of the two same-profile terms the models write in each family. The per-term rates predict shares of 5.6% and 17.2% for commitments with all four terms. The observed shares are 5.7% and 21.2% (Appendix K.5).

A different reader removes the gap. We agree. The true-set selector result is now in the main text (Section 5), next to the other readers.

Thank you again. Your review improved the paper.

AuthorsOpenReview ↗

Follow-up to Reviewer XBFn: A Direct Test of Weakness 1

Thank you again for your careful review. We add one new analysis that tests your Weakness 1 directly. It is in an updated revision (Section 5 and Appendix K.4).

Can the runs separate overlap from what the report says?

You noted that the design cannot separate an effect of overlap from the report being more informative at high L. The saved runs can do this, because what the report says differs across families at the same level. We measure it by the slot reader's gap, which uses only the two reports. We regress the models' gap on the reader's gap and on L, over the 72 family-level cells (three levels, 24 families; family bootstrap).

Terms in the fitCoefficient on LCoefficient on reader's gapR²
L only+0.419 [+0.325, +0.508]0.54
Reader's gap only+0.90 [+0.79, +1.00]0.77
Both+0.010 [−0.122, +0.138]+0.88 [+0.66, +1.11]0.77
  • With the reader's gap in the fit, L adds nothing.
  • At a fixed level, the reader's gap still predicts the models' gap: +0.86 [+0.63, +1.09].
  • The same holds over all six levels. There the coefficient on L is +0.031 [−0.061, +0.117].

So overlap acts through what the report says. This agrees with the reading in your review (Strength 2), and the paper now states it with this test.

A check at equal overlap. The profile-grouped level has the same L (0.78) and the same number of answers as the four-term core, but a different report. Between the two levels, the models' gap changes by −0.032 [−0.120, +0.053], and the slot reader's gap changes by −0.032 [−0.095, +0.033].

A shared concern. Your Weakness 2 (the catalogue reader) is the same concern as Reviewer xXyo's first weakness. Our reply to Reviewer xXyo answers it with the same evidence (Table 3 and Section 5). Reviewer xXyo has since replied and raised their rating, so that thread may also be useful.

We hope the revision resolves your two weaknesses. If any point remains open, we will be glad to answer it before the discussion closes. If it does resolve them, we would be grateful if you would consider updating your rating.

Thank you for your time and consideration.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response and Summary of Revisions

We thank all three reviewers for their careful and constructive reviews. We have uploaded a revised paper. All changes are highlighted in blue. The study, the data and the primary analysis are unchanged.

Summary of changes

  1. Claims (abstract, Section 1). The report lists the most frequent components, so the direction of the effect follows from the report's design. The paper now states this. It measures the size of the effect and how models use the report.
  2. Section 4. We add a full test against no report, and we test each step of the dose-response.
  3. Section 5 (rewritten). It now reports how models follow the report, mechanical readers with and without the catalogue, and the step at which accepted answers are lost.
  4. Section 6. It now starts with the study of feedback on models' own SAT answers, with results for each model.
  5. Section 7. It discusses the authored acceptance sets and practical use.
  6. New Figure 2. It shows two reports exactly as the models received them.

The added analyses use the saved runs. They are collected in the new Appendix K.

Numbering. The main results figure is now Figure 3 (Figure 2 in the submission). Tables 1 to 5 keep their numbers. Table 6 is new. Appendix tables shift up by one. Appendices A to J keep their letters.

We reply to each reviewer below.

AuthorsOpenReview ↗

Summary of the Discussion

We thank the reviewers and the Area Chair. This note summarises the discussion and the state of each concern.

What the paper claims after revision. The paper is a controlled measurement of how a property of the acceptable-answer set, its overlap, governs what a component report is worth to language models.

Points the reviewers agreed on. All three reviewers noted the controlled design, which holds the data, the number of answers and the report format fixed. Reviewers also noted the no-report controls, the separate graded and exact outcomes, and the range of models. Two reviewers had no substantive concerns about soundness, and the third had minor or fixable ones.

Main concerns and how the revision answers them

  1. The effect follows from how the report is built (XBFn W1; KY9d W2, Q2). We agree, and the abstract now says so. The revision measures what the design does not fix. Models follow reported slots without weighting them by stated frequency. A reader that only fills slots tracks the models' gaps. A new regression separates the two factors: with the reader's gap in the fit, the coefficient on overlap is +0.010 [−0.122, +0.138] (Section 5, Appendix K.4). Against no report, the interaction is +0.484 [+0.389, +0.582], and each step of the dose-response is tested (Section 4).

  2. The exact-recovery gap relied on a reader that sees the catalogue (XBFn W2; xXyo W1, Q1). We agree, and we removed that conclusion. Section 5 now shows that about three fifths of the gap is catalogue access (−0.334 with the catalogue, −0.126 without). It also locates the loss in the models' own answers. Models commit the right term in 91% of slots whose reported profile is unique, and in 41% of slots where two terms share the profile.

  3. External evidence (xXyo W2, Q2; KY9d W3). Section 6 now starts with a prospectively specified study of feedback on models' own SAT answers. The slope is +0.620 [+0.434, +0.806] against a sham and +0.609 [+0.417, +0.815] against no report. The two models excluded by the ceiling rule show the trend on harder formulas (+0.999 [+0.694, +1.303]).

  4. The acceptable sets are authored, and practical use (KY9d W1, Q1, Q4). The sets are authored to hold the number of answers fixed. The revision shows what real criteria give. Fit alone gives L = 0.03, and a parsimony criterion gives 0.71. On published benchmarks, L = 1 for all measured AI-Feynman problems (15) and dynamical-system equations (102), and L is at least 0.9 for 20 of 23 causal networks (Appendix K.9). Real benchmarks therefore lie at the high-overlap end, where the report gains most (+0.507 over no report at L = 1). They do not vary in overlap, so a controlled comparison needs authored sets. Overlap can be estimated from five sampled answers (Section 6.3).

  5. Presentation (all reviewers). A new Figure 2 shows two real reports. Section 5 is reorganised, and Section 7 states the contribution in one sentence.

State of the discussion

  • Reviewer xXyo replied that our response addressed their concerns and raised their rating to weak acceptance.
  • We posted a short follow-up to Reviewers XBFn and KY9d, with the new analyses in points 1 and 4.
  • One requested experiment, giving the models the catalogue in the prompt (xXyo Q1), could not be run in the response period. We answered it from the reader side (point 2).

Revisions. The first revision was uploaded on 15 November. A second revision, uploaded with the follow-ups, adds the regression (Section 5, Appendix K.4) and the benchmark measurements (Section 7, Appendix K.9). All changes since the submission are shown in blue text. The main text is 10 pages.

The reviews and the discussion made the paper clearer and more precise, and we are grateful for them. Thank you for your time and for handling our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.