There is no new method or metric. Retention is close to the known negative-flip idea from model-regression work. The main contribution is organizing these existing comparisons into an audit framework, but the methodological novelty is limited.
Several analyses that are standard in related work are missing. Selective comparisons are made at a single coverage determined by the model's abstentions, rather than using full risk-coverage curves ranked by model confidence (e.g. token probabilities). Prompt sensitivity is tested with only one revision, although that revision changed retention substantially (1/14 to 14/14 pairs on clear floral). Voting is not compared with standard position-debiasing methods such as PriDe (Zheng et al., 2024) or contextual calibration (Zhao et al., 2021).
The results are mostly null: no model demonstrates added value over the score rule, and the best case is reproducing it. This is expected and partly built into the setup. Thinking is disabled, output is capped at 64 tokens, only the first line is parsed, and the prompt tells the model not to output an explanation (Appendix F). With no room to reason, the models mostly follow the larger score, even in conditions where SMILES and dispersion are also shown. The setup therefore does not test whether more capable or reasoning-enabled models could extract more value from the evidence.
The main retention finding raises concerns about position bias. All 24 unstable pairs choose position A twice (Table 2), and in the supplementary material, the 11 pairs LF loses have smaller score gaps. The difference is real (11 vs. 0 discordant pairs), but the observed pattern is consistent with position-dependent selection and does not establish that the effect reflects a new or distinct failure mode.
The models are small and narrow. All are 1.7B-8B plus one 32B, with no frontier or reasoning-enabled models.
The predictor is custom (Morgan fingerprints + MiniLM) rather than an established one such as POM/OpenPOM, which makes the results harder to compare with other olfaction work. A custom predictor is fine for a controlled audit, but the results should also be shown with an established one to assess generalizability.
The samples are very small. The main released tasks have 101 pairs (nutty has only 17), and there are only 18 clear floral human pairs. In the 32B clear-pair risk comparison, the model answers 22 floral, 5 roasted, 6 meaty, and 0 caramellic presentations. The results for roasted and meaty are particularly sparse, and the absence of errors in these small samples does not establish equivalence or reliable risk guarantees.
Most of the study is post hoc and exploratory. Follow-up protocols were frozen after earlier results were inspected, and the wording revision was motivated by observed behavior. An independently sampled, pre-specified evaluation is needed to show whether the reported patterns persist beyond the cases that motivated the analyses.
Intervals are unadjusted for multiple comparisons.
There is no learned baseline on the same evidence. Without one, it is difficult to distinguish whether the lack of added value reflects limitations of the LLMs or the limited information in the evidence. More broadly, the experiments do not establish whether the observed limitations arise from the evidence, the interface, or the model's capabilities.
The paper is hard to follow, starting with the abstract and introduction. The abstract does not clearly state the task (choosing between two molecules given predictor scores). It lists raw counts (158/202, 67/78, 18/63) before the reader knows what they refer to, and it does not summarize the main finding in plain terms: the LLMs at best reproduce the score rule and do not demonstrate added value beyond it.
The introduction gives only one sentence of motivation for placing an LLM on top of an odor predictor. It also uses terms like "retention opportunities", "correction opportunities", "evidence reassignment" and "supplied-target reading" before they are defined. The rest of the text is very dense.