I have a serious concern about the factuality of the reported GPT-2 results. This is in the territory of possible academic fraud, and is by far a bigger concern than any other weaknesses I have about the strength of results (which I list separately under questions for authors).
I initially became suspicious due to a strange phrasing of the reproducibility statement. I then asked GPT-6 to look through the paper for other similar statements, and it found a rather long list of very suspicious statements throughout the appendix.
I think the GPT-6 reponse is pretty clear and concise; for full transparency, I am pasting the un-edited (but truncated) reponse at the end (the un-truncated version is in "AI Documentation For Reviewing"). I have manually verified that none of the listed items are hallucinated.
Due to all these suspicious parts, I am recommending strong reject unless the authors can provide code that one can independently run and verify that the claimed GPT-2 results are reproducible.
my prompt:
Attached is a paper that I am reviewing. I suspect that the authors fabricated their data, but some of the llm-written sections seem to let-slip this fact (e.g. ". The accompanying numerical scenario file fixes every assumed adapter effect and approximation error and checks their normalizations and ordering. These inputs are not extracted pretrained measurements")
I need you to carefully review the text and data and list the evidence for and against my suspicion.
GPT-6-Astra-max response
Your concern is strongly supported for the claimed GPT-2 experiments. The paper explicitly describes those numbers as assumed and the evaluation as unexecuted, while presenting the same numbers as empirical findings in the abstract, introduction, figures, and conclusions. That is a substantial reporting problem.
The narrower conclusion is well supported: constructed numerical scenarios are presented as measured GPT-2 results. The PDF alone does not establish that every experiment was fabricated or establish the authors’ intent.
I reviewed all 26 pages and independently checked several numerical relationships in 26965_When_Can_Prefixes_Compil(1).pdf. References below use the paper’s printed page and line numbers. I did not have the accompanying code, scenario file, checkpoints, or raw outputs.
The evidence supporting your concern is as follows.
1. There is an even more explicit admission than the sentence you identified: the GPT-2 evaluation is described as “unexecuted.”
Appendix F.5, p. 19, lines 998–1000, discusses the evaluation in Appendix J and ends:
“Its unexecuted status is distinct from the controlled results in Section 7.”
Appendix J then begins, p. 23, lines 1207–1208:
“Adapter effects, conic brackets, optimizer errors, and transfer errors are assumed; their normalizations and order relationships are calculated.”
This is unusually decisive evidence. It does not merely say that the experiment uses synthetic inputs or chosen hyperparameters. It says that the quantities being reported as experimental outcomes—including optimization and transfer errors—are assumed.
Your quotation from the reproducibility statement, p. 10, lines 490–492, independently reinforces this:
“The accompanying numerical scenario file fixes every assumed adapter effect and approximation error and checks their normalizations and ordering. These inputs are not extracted pretrained measurements.”
Together, these passages strongly support the conclusion that Appendix J reports hypothetical outcomes rather than results extracted from executed GPT-2 experiments.
2. The main text repeatedly presents these same hypothetical numbers as empirical findings.
| Location | Presentation in the paper | Why it matters |
|---|
| Abstract, p. 1, lines 25–28 | At three GPT-2 heads, the capped optimum leaves 18.4%–74.2% of the adapter effect uncompiled. | Presented as a finding about actual pretrained heads, without a hypothetical qualification. |
| Introduction, p. 2, lines 79–81 | The test leaves that fraction of “trained adapter effects” uncompiled. | Implies that trained adapters were obtained and evaluated. |
| Figure 1, p. 2 | The GPT-2 numbers appear under “Measured”; the caption says “Measured values are from Section 7.” | Explicitly labels the numbers as measurements. |
| §7.4, p. 9, lines 452–461 | “Finally, we apply the capped test…” followed by specific residuals, ten-restart comparisons, and held-out transfer errors. | Describes completed experimental procedures and their outcomes. |
| Conclusion, p. 9, lines 479–483 | Value–query ordering changes across pretrained heads; the test separates resource limits from optimizer error. | Draws substantive empirical conclusions from the assumed numbers. |
This is therefore not an isolated ambiguous sentence. The contradiction affects a highlighted contribution and its supporting evidence.
3. The appendix says that a central empirical pattern was deliberately built into the scenarios.
Appendix J.2, p. 23, lines 1239–1241, states:
“These scenarios deliberately include both orderings of value and query targets.”
It gives the same head-0 and head-4 examples that §7.4 presents as observations, then says:
“Such a pattern would be compatible with a target-specific boundary.”
The distinction is consequential:
- A scenario deliberately chosen to exhibit both orderings illustrates what the theory permits.
- An experiment finding both orderings would provide evidence that this happens in pretrained models.
The main text treats the former as the latter. The claimed head-dependent ordering is therefore not independently demonstrated by these tables.
4. The numerical tables contain repeated arithmetic patterns consistent with constructed scenarios.
I recalculated the following differences from the printed values:
| Data | Repeating pattern |
|---|
| Table 8, p. 25: normalized best-learned error minus conic upper endpoint | 0.025, 0.030, 0.035, 0.040, repeated for each of the three heads, to displayed precision. |
| Tables 7 and 9: evaluation-oracle ratio minus fitting-domain conic ratio | 0.025, 0.028, 0.031, repeated four times in printed row order. |
| Table 9: conic-transfer ratio minus evaluation-oracle ratio | 0.065, 0.066, 0.067, repeated four times. |
| Table 9: learned-transfer ratio minus conic-transfer ratio | 0.047, 0.049, 0.051, repeated four times. |
| Table 10, p. 26: learned error minus exact-count optimum | 0.029, 0.034, 0.058 for slot counts 1, 4, 16, repeated across all six target/cap combinations. |
For example, all 12 rows of Table 9 can be reconstructed from the fitting-domain ratios by adding three short, repeating sequences.
Such regularity alone would not prove fabrication. Here, however, it corroborates the explicit description of a scenario file that fixes errors and checks their ordering. In particular, the small optimization and transfer gaps highlighted in §7.4 appear to be features of the assumed scenarios, not established properties of actual runs.
5. The apparent solver certificates are explicitly disclaimed.
Table 7, p. 24, lines 1243–1245, presents lower and upper conic endpoints separated by at most 10−6. This gives the appearance of tightly resolved numerical optima.
But Appendix J.5, p. 25, lines 1331–1333, says that a lower endpoint requires a checked infeasibility witness and then states:
“The intervals in Table 7 are not such witnesses.”
Appendix J.2, p. 23, lines 1236–1238, also says:
“A physical reconstruction attaining the upper endpoint would be obtained from Equation (15), but its pretrained attention evaluation is not asserted by the assumed table.”
Thus, the paper does not establish either that these brackets certify the actual GPT-2 optimization problems or that the corresponding prefixes were reconstructed and evaluated.
The mathematical theorem can justify a procedure for computing an optimum. It cannot establish that an assumed number is the optimum for a particular pretrained model.
6. Important provenance for the supposed trained-adapter experiment is missing from the PDF.
Appendix J gives experimental-looking details: heads, ranks, training seeds, context lengths, ten restarts, 2,000 optimization steps, and a learning rate. However, I could not find:
- An identifiable trained-adapter checkpoint or the adapter training objective and dataset.
- The actual fitting/evaluation contexts or their source.
- An explanation of how the three stated training seeds produce the single reported row per head/target/rank.
- Actual extracted summaries, solver residuals, or reconstructed-prefix outputs.
These omissions alone would be a reproducibility weakness, not proof of fabrication. In combination with “unexecuted” and “assumed,” they reinforce the reading that this is a proposed experimental design populated with illustrative outcomes.
There is also a minor cross-table inconsistency. For head 0, value rank 4, Table 8’s learned error normalized by Table 7’s adapter effect is approximately 0.437, whereas Table 10 reports 0.441 at the matching caps and four slots. For head 4, query rank 4, the corresponding values are 0.359 and 0.353. Separate optimization runs could explain this, so I would treat it as a clarification issue rather than central evidence.
my follow-up prompt to clarify the patterns in tables 7 and 9:
can you double check your claim about numerical tables. I cannot verify the four-time repetition in tables 7 and 9
GPT-6-Astra-max response
I rechecked it: the arithmetic is correct, but I should have made the comparison clearer. The repetition is in the differences, taken in printed row order. The three-row cycle crosses the boundaries between heads.
Specifically, subtract:
- A: Table 7’s last column, Ehi/DG, on p. 24.
- B: Table 9’s “Evaluation oracle” column, on p. 25.
Here is every subtraction from the PDF: 26965_When_Can_Prefixes_Compil(1).pdf
| Row | Head | Target / rank | Table 7: A | Table 9: B | B−A |
|---|
| 1 | 0 | Value / 1 | 0.184 | 0.209 | 0.025 |
| 2 | 0 | Value / 4 | 0.407 | 0.435 | 0.028 |
| 3 | 0 | Query / 1 | 0.631 | 0.662 | 0.031 |
| 4 | 0 | Query / 4 | 0.542 | 0.567 | 0.025 |
| 5 | 4 | Value / 1 | 0.683 | 0.711 | 0.028 |
| 6 | 4 | Value / 4 | 0.742 | 0.773 | 0.031 |
| 7 | 4 | Query / 1 | 0.267 | 0.292 | 0.025 |
| 8 | 4 | Query / 4 | 0.319 | 0.347 | 0.028 |
| 9 | 8 | Value / 1 | 0.388 | 0.419 | 0.031 |
| 10 | 8 | Value / 4 | 0.566 | 0.591 | 0.025 |
| 11 | 8 | Query / 1 | 0.451 | 0.479 | 0.028 |
| 12 | 8 | Query / 4 | 0.487 | 0.518 | 0.031 |
Thus, rows 1–3, 4–6, 7–9, and 10–12 each give the sequence (0.025, 0.028, 0.031). It does not restart at each head.
The other two patterns occur entirely within Table 9:
| Rows | Conic transfer − Evaluation oracle | Learned transfer − Conic transfer |
|---|
| 1, 4, 7, 10 | 0.065 | 0.047 |
| 2, 5, 8, 11 | 0.066 | 0.049 |
| 3, 6, 9, 12 | 0.067 | 0.051 |
For example, its first row gives 0.274−0.209=0.065 and 0.321−0.274=0.047; its fourth row gives 0.632−0.567=0.065 and 0.679−0.632=0.047.
Two qualifications matter. These equalities hold at the displayed three-decimal precision. Also, Table 7 and Table 9 normalize by their respective fitting- and evaluation-domain adapter effects, so the first comparison concerns normalized ratios, not an additive increase in raw error. The repeating pattern is real, but the explicit statements that the outcomes are “assumed” and the evaluation “unexecuted” remain much stronger evidence about their provenance.