- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The author found that in the field of AI for Research, even when experimental requirements are repeatedly included in the prompt, agents still quietly alter the experimental conditions. The author refers to this phenomenon as methodological hallucination, categorizes the different types of distortion, and proposes a method called ABE-Ralph. This approach requires the model to clearly write out the experiment first before checking the execution outputs, and small-scale experiments are conducted to demonstrate the method's effectiveness to a certain degree.
Critical strengths
The author's observation is meaningful.
Critical weaknesses
- The method claimed by the author does not actually work in practice. One of the three checks designed by the authors was never actually executed, specifically the L3 critical modules. They didn't fill in the required critical modules in any of the thirty experiment runs, so they all defaulted to passing. In other words, the authors completely failed to implement what they claimed to do in their own method. It looks like they didn't even check their own work before submitting it so irresponsibly.
- The experimental setup used by the authors may invalidate their claim, as all experiments were run on a single 4070 GPU. It is possible that the limitations observed in the models were entirely hardware-driven. The authors had the models perform various tasks involving language models like BERT, as well as other generative models such as DDPM and DDIM. This raises concerns about whether these architectures can theoretically even run properly on an RTX 4070. The authors claim to have identified problems in the agent, but in reality, this failure to complete the tasks could simply be due to hardware constraints.
- The writing in this paper is very poor. For instance, Figure 1 is extremely small, and the text on it is completely illegible. Furthermore, the various bar charts later on are inconsistent in style and have different background colors, as if they were screenshotted from somewhere else. There are also some obvious typos. For instance, the percentage sign is written twice everywhere. The author didn't even bother to read through their own paper.
- The authors didn't run experiments on recognized or complete datasets. Therefore, the paper lacks a complete evaluation and fails to support any of the claims the authors are making.
Questions for authors
NA
4 comments
Response to Reviewer XqXf (2/3)
P2 — "everything ran on one 4070", and what was actually run
Half right, and the wrong half is ours. Every run executed on one RTX 3080 with 12 GB. The hardware check at the head of the run log prints NVIDIA GeForce RTX 3080, total_memory: 12.0 GB, compute capability 8.6, 20 CPU cores (bm_006_lora_lora/run_all.log:24-31; the same block appears in ten workspaces spanning four batches), and in bm_012 the agent's own experiment_state.json records NVIDIA GeForce RTX 3080 (12 GB VRAM) — exceeds the plan's RTX 4070 Laptop (8 GB). The RTX 4070 Laptop / 8 GB line in all 30 contracts is the budget the agent was given, not the machine it ran on; §5.1 reproduced it as a description of the host.
Erratum: the sentence should read "a single RTX 3080 (12 GB VRAM), with per-run time budgets drawn from the reference contracts (median 27 h, range 10–58 h)" — the recomputed median is 27 h, not 26. The report lines naming the 3080 were right, and we had read them as an error. Neither verify_reproduction.py nor alignment_verifier.py inspects hardware, which is how an unverified configuration value reached the settings paragraph — the failure the taxonomy exists to catch, in our own paper.
The argument is unaffected. On one consumer card, ViT from-scratch pre-training, ImageNet-scale training and MuJoCo control stay out of reach; the ViT run stays M2+M3+M5 (trained from scratch, scale-dependent conclusion, neither declared). The card was binding: the PEGASUS report lists Constrained batch size as the price of running on RTX 3080 12 GB (bm_008).
The clock, by contrast, was never close. Nine of the 30 reports state how long the run took, and every one finished well inside its contract budget.
Table 7 — contract compute budget against the runtime the run itself reports
| Run | max_gpu_hours | Stated runtime | Share used |
|---|---|---|---|
bm_008 PEGASUS | 36 | 67 min | 3.1% |
bm_014 CLIP | 12 | 27 min | 3.8% |
bm_019 YOLO | 18 | 0.75 h | 4.2% |
bm_020 PatchCore | 24 | 60 min | 4.2% |
bm_006 LoRA | 24 | 77 min | 5.3% |
bm_009 LED | 48 | 6.5 h | 13.5% |
bm_005 RoBERTa | 24 | 4.5 h | 18.8% |
bm_013 ViT | 30 | 8.5 h | 28.3% |
bm_015 SimCLR | 36 | 12 h | 33.3% |
The median run used 5% of its budget and the largest share 33%: M5 records runs that stopped incomplete rather than runs that ran out of time. The other 21 reports state no runtime, so this is a lower bound on coverage, not a summary of the corpus.
Table 8 — what each labelled run executed, as its own report states it
| Run | Label | Paper's setting | Executed | Where its own report says so |
|---|---|---|---|---|
bm_026 PPO | M2+M5 | nine MuJoCo continuous-control tasks | CartPole-v1 (discrete), 3 seeds, 100K timesteps | Summary; the conclusion speaks of "continuous control tasks" without the substitution |
bm_004 BERT | M2+M5 | full GLUE suite | SST-2, MRPC, QQP; ≥3 epochs | header; §1 |
bm_015 SimCLR | M1+M2 | ResNet-50, batch 4096, 800 epochs | ResNet-18, batch 256, 400 epochs on CIFAR-10 | header table; §5 |
bm_003 ColBERT | M2+M5 | ~200K training steps | 500 steps, 500 queries, 82K passages | header; gap attributed to "reduced training scope" |
bm_008 PEGASUS | M2+M3+M5 | 287K CNN/DailyMail examples | 3,000 samples, 2 epochs (1.2%) | §3 table; §8 says absolute ROUGE not reproduced |
bm_013 ViT | M2+M3+M5 | ViT-B/16 pretrained on JFT-300M, fine-tuned | ImageNet-pretrained at 93.29%, from scratch at 26.02% | §2 results table; §7 |
bm_017 U-Net | M3+M5 | full ISBI 2012 (30 images) | 30 epochs on 24 training images (6 held out) | §4; §6 qualifies the claims by scale |
bm_028 GraphSAGE | M2+M4+M5 | 15 epochs, hidden 256, batch 512 | 5 epochs, hidden 64 | §5; skipped conditions listed |
These are all the runs carrying M2 or M3 except bm_018, which produced no report. The scale is recoverable from the report in all eight cases; what varies is where it sits. PEGASUS and GraphSAGE put it in the conclusions; PPO names the environment only in the summary line and then generalises to "continuous control tasks".
Response to Reviewer XqfX (3/3)
P3 — Figure 1 and formatting
Figure 1 is a 1487×578 raster, set at 0.62\textwidth in the submission, which puts its labels near 4–5 pt. In the rebuilt PDF it runs at full width and they are legible; a revision would re-export it at ≥8 pt. We rebuilt the PDF and fixed the typos.
P4 — datasets used at reduced scale
The canonical datasets are used at reduced scale: GLUE, NaturalQuestions, TriviaQA, MSMARCO, CIFAR-10/100, ImageNet subsets, ISBI, PASCAL VOC, Cora/Citeseer, MuJoCo. The reductions are the object of study, not an accident, and Table 8 gives that account for every run carrying M2 or M3.
Ethics flag
We take the flag seriously. It rests on the concern that results were reported that the authors did not produce. Each factual claim in the review is answered against a file in the archive that Table 5 inventories, and Appendix F reports the runs where our own pipeline failed: PEGASUS −32% against BERT, and PPO claiming continuous-control success while none of the nine MuJoCo runs completed. Those are in the labelled corpus with the same labels we apply to the agents. We will answer any specific question about any artifact.
Thank you for the response. The rebuttal confirms that L3 was never effectively executed and that the hardware setup was misreported. These issues undermine the reliability of the audit itself. I will keep my score.
Response to Reviewer XqXf (1/3)
Thank you for auditing the audit.
P1 — none of the three checks was ever executed
L1 and L2 ran; L3 did not. L1 wrote a verdict in 47 workspaces of the archive, 23 of the 30 benchmark runs among them; across the 47 the split is
MET_ALL8,MET_PARTIALLY29,FAILED9,UNKNOWN1. L2 wrote 11 review logs, 6 with a parseableDECISION. L3 was vacuous because no contract declarescritical_modules— a field that is optional in the schema and was omitted by the generator in all 30.Table 5 — what the archive carries, per benchmark run
bm_018,bm_022,bm_024,bm_025,bm_030bm_018,bm_022,bm_024,bm_025,bm_028,bm_029,bm_030results/metrics.jsonbm_019–bm_021,bm_022–bm_025,bm_027,bm_030METRICS_MISSINGand is skippedbm_0032/3,bm_0072/3,bm_0161/2bm_008, one inbm_026, two inbm_018, which has no reportbm_006passes this test while its headline number contradicts the fileThe row is loose by construction — a match on any spelling from one to four decimal places counts — and we flag it as the one number here that flatters us.
With
critical_modulespopulated from the papers and run on the stored code, L3 does no better than vacuous, and the reason is scope rather than absence of components.Table 6 — L3 with
critical_modulespopulated, 20 runs and 62 components<workspace>/main.pyonlyThe component code is there; the check looks in the wrong place. Nine names appear in no file at all —
DualEncoder; BERT'sTransformerEncoder,MaskedLMandNextSentencePrediction;GapSentenceGeneration;DilatedAttention;ContrastiveLoss; and U-Net'sContractingPathandExpandingPath— so name normalisation is needed as well as scope. A revision would either fix both or remove L3 from the protocol.