- Rating
- 1
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes an LLM mid-training method where instead of instruction-tuning on generic natural language data, they train on synthetic deductions in propositional logic. Their method results in increased downstream performance on various benchmarks for a variety of models at differing scales.
Critical strengths
- The method is simple and easy to understand. The fact that a simple intervention can yield non-trivial improvements to performance is noteworthy.
- The number of benchmarks, model families, and scales evaluated are numerous and makes progress toward supporting the claim that this approach can be used as a general mid-training method.
Critical weaknesses
-
The paper's presentation is poor. The various experiments, particularly the numerous ablations, need to be motivated better so that the scientific questions they are answering are clearer. For example, it is unclear what the purpose of the 'shuffled lines' and 'derivations alone' ablations are as results are presented with little to no exposition.
-
The choice of baseline mid-training data content and scale is unmotivated and lacks consideration. Comparisons against baselines trained on 100k examples randomly sampled from general instruction-tuning corpora do not sufficiently control for task specific over-fitting of the proposed method. It is possible that the downstream benchmarks share superficial similarities with the injected synthetic data.
Questions for authors
-
Can the authors provide evidence that the proposed method does not simply fine-tune LLMs for the specific tasks that they are being evaluated on? For example, does your proposed mid-training method yield improved performance on tasks that do not have similar syntactic structure?
-
Can the authors provide stronger evidence that the method they propose does not harm the LLM's performance on non-reasoning tasks far removed from symbolic reasoning (e.g. language modelling, general question answering in non-technical domains). Otherwise, it is hard to view this method as something other than domain specific fine-tuning.
4 comments
As the discussion period ends soon, we would be grateful to hear whether our responses and the revised PDF (changes marked in blue) address the reviewers' concerns, and which points remain open, so that we can still respond to them.
Thank you for your response. I now have a better understanding of the details pertaining to the issues that I raised in my original comment, and I raise my confidence.
Thank you for your reply. We noticed that the confidence was raised to 3 while the rating remained at 1, the lowest rating on the scale, although in our rebuttal we addressed all the points you raised. Since your comment suggests that the concerns have been clarified, we would like to ask whether the rating is intended as it stands, or whether it should also be updated. If any point is still open, we would be grateful to know which one, so that we can address it before the end of the discussion period.
We thank the reviewer for the careful review and for recognising the simplicity of the method and the range of models, scales and benchmarks. Whether the gains come from fine-tuning on data that resembles the benchmarks is central to our claim as well. To test it, we trained four content ablations for two derivation mixtures (eight new models) and evaluated all models on further benchmarks far from symbolic reasoning.
The training problems share little surface form with the benchmarks. They are chains over constants c0 to c25 with colour, direction and tree words as predicates, contain no negation and ask which of four candidates holds. None of the benchmarks uses this format. Accuracy nevertheless rises on BIG-Bench Hard tasks posed as English stories, by 12.5 points on web of lies and 8.1 on object tracking. If the gains came from superficial similarity, the tasks closest in form to logic problems would be expected to gain most, yet LSAT-AR, LSAT-LR, LogiQA and MMLU formal logic change by -1.9 to +1.8 points. The tasks that improve share a requirement, namely carrying a truth value or a state along a chain of steps, where the baseline often loses an intermediate value (Section 6.2).
The content ablations replace the same rows with other content. Procedural traces of column addition, sorting, base conversion and run-length encoding contain no logic and nothing resembling ProofWriter, yet recover most of its gain (+8.5 against +11.5 at 25 percent). This part of the gain therefore comes from step-by-step traces in general and cannot be explained by similarity to the benchmark. Two gains require the deduction content, on held-out generator problems (+3.8 and +9.5 against +0.2 and -0.1) and about half of the web of lies gain (+12.5 against +5.9).
The baseline is chosen to answer one question, namely what happens when part of a fixed instruction-tuning budget is spent on generated deductions. All mixtures therefore have the same 100,000 examples and one epoch, so that differences can be attributed to the content of the replaced rows. Appendix I varies the size from 25,000 to 200,000 examples. Section 5.3 repeats the comparison on Tulu 3, whose baseline answers "false" on 83 percent of ProofWriter items against 73 percent "true" for Dolci. The derivation procedure is learned on both corpora (66.4 percent of held-out derivations accepted on Tulu 3 against 0.0 for its baseline), while the ProofWriter and FOLIO gains are smaller on Tulu 3, whose baseline is already 14 points higher on ProofWriter.
We evaluated language modelling and question answering far from symbolic reasoning (Appendix K). On BoolQ, SciQ, TriviaQA, Natural Questions, LAMBADA, HellaSwag, PIQA and WinoGrande the derivation mixtures stay within 0.9 points of the baseline, and WikiText word perplexity changes by less than 0.05. No general benchmark falls by 4 points or more. The largest drop, 3.9 points on HumanEval at 25 percent, also appears under every content ablation, so we attribute it mainly to the code examples removed with the replaced rows.
We revised the presentation throughout, and all changes in the revised PDF are marked in blue. A new method section (Section 3, Algorithm 1) now describes the generator, the renderings, the checker and the mixing procedure, every number in the text was checked against the tables, and each ablation in Section 5.3 now begins with its purpose. Shuffled lines keep every token, the length and the conclusion of each derivation and destroy only the order of the steps. They test whether the model learns from the sequence of inferences or only from the facts and answers. The object-tracking gain disappears (+1.6 against +8.1) while the ProofWriter gain remains, so state tracking depends on the order of the steps and the ProofWriter gain mostly does not. Derivations alone (Appendix D.6) test whether instruction data is needed to apply the derivation skill to other tasks. Without instruction rows the model scores zero on every benchmark, so the transfer requires both.
We hope that this evidence addresses the reviewer's concerns. Should further evidence be needed for the reviewer to reconsider the assessment, we would be grateful to learn which experiments would be most informative, so that we can run them during the discussion period.