acceptodds
Desk rejected at ICLR 2027

Evidence Response Is Not Decision Retention: An Audit of an Olfactory LLM Interface

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

We present a decision-level audit of predictor-backed language interfaces, separating behavioral response, assignment advantage, retention, and added value. We apply it to a fixed olfactory predictor, using released odor annotations and, separately, human applicability ratings. On 101 floral annotation pairs, two complete Qwen3-4B interfaces both make 158 correct choices across 202 presentations, yet retain 67/78 and 78/78 rule-correct pairs in both orders. A fixed vote across four label–position encodings recovers all 78 for the first interface without correcting any rule error relative to the annotations. Voting is not uniformly beneficial: on odorless pairs, it makes 18 errors among 63 accepted pairs, compared with 10 for numerical gap selection at the same coverage. In the separate human-rating task, Phi-4-mini and Qwen3-4B reproduce the score rule on every clear floral presentation under revised wording. One 32B configuration correctly reads all 160 supplied reference-rating presentations, but shows no observed matched-coverage risk advantage over numerical selectors on clear pairs when given predictions. The audit distinguishes recovery of correct decisions from improvement beyond the visible-score rule.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, computational olfaction, empirical evaluation, evidence use, decision retention, order sensitivity, selective prediction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jqzCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors investigate the idea of using an LLM to enhance the predictions of a pre-trained olfactory predictor scores. Although its an interesting idea, the paper is an aggregation of half-baked results obtained by more likely an auto research run.

Critical strengths

The direction of using an LLM to correct the predictions is sound and much needed.

Critical weaknesses
  1. Half-baked experiments
  2. Motivation is not structured
  3. Task setup is hard to follow
  4. The choice of models and the experiment design is naive.
Questions for authors

Please spend some time writing the paper, it looks completely an auto research run which is poorly motivated.

Reviewer_xKbTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether a small LLM (Qwen3 1.7B-8B or Phi-4-mini) placed on top of a fixed odor prediction model improves pairwise decisions (i.e., which of two molecules is more associated with a given descriptor?) beyond simply selecting the molecule with the higher predicted score. The predictor is an ensemble of five networks combining Morgan fingerprints with MiniLM embeddings of descriptor names, trained on a GoodScents-Leffingwell (GS-LF) with 28 descriptors withheld from training. Evaluation uses OdorNet annotations (mainly floral, with odorless and nutty in some analyses) and human panel ratings from the Principal Odor Map (POM) data (floral, roasted, caramellic, and meaty). The main findings are that two prompt variants with equal accuracy retain different sets of correct decisions and that majority voting over four label/order encodings recovers the score-based decisions but corrects none of the predictor's errors.

Critical strengths
  • Comparing the LLM against the decision based on the predicted score is the appropriate baseline and is often missing in studies of LLMs built on top of predictors or tools.

  • The observation that equal aggregate accuracy can conceal lost correct decisions is valid and illustrated by the pair-level transition analysis (Table 2).

Critical weaknesses
  • At the current state, the paper is very hard to follow for a reader, and I had to re-read entire paragraphs to understand them. Even the abstract and introduction are hard to parse without already knowing the paper. To name just a few, in the following is a non-exhaustive list of examples: a) "Correctness" used in Eqn. 1 (line 116) is never explicitly defined, and from the context it seems it is the standard accuracy, with TIEs counted as incorrect. b) What is the definition of "direct evidence"? It is used on line 109 but only named on line 189 and defined in Appendix B. c) BihB_{i}^{h} is redundant since Hi0=Hi1H_{i0} = H_{i1} for all i. d) There are non-standard terms such as "weight-fitting", "affine-adjustment", and "development" sets, which appear to correspond to training, calibration, and held-out validation sets, respectively e) E1/E4 in Figure 2 are only defined on page 33 in Appendix K, but if I understand well, these correspond to r=0 and r=1 in Eqn. 1. It would be more clear for the reader to use consistent naming.

  • The core hypothesis that I understood from the text is whether adding an LLM readout on top of the dedicated odor prediction model can improve on selecting the molecule with the higher predicted descriptor score. While the authors' own answer is mostly "no", the motivation of studying this hypothesis is elusive. In order for this hypothesis to be true, the LLM and the dedicated scoring model would need to have some complementary information or skills, and it is not clear what additional skills the LLM provides here. The only additional inputs are the SMILES string, which LLMs are not known to parse reliably, and the LLM’s prior knowledge, which for genuinely novel molecules is limited. Therefore, could the authors clarify the motivation and the expected mechanism of improvement that they are testing for?

  • The novelty of the paper is limited. It combines known evaluation techniques (negative-flip analysis, counterfactual evidence swaps) and applies them to a narrow olfactory setting.

  • The majority of analyses are limited mainly to floral descriptor for the database-label tasks and 4 descriptors (floral, roasted, caramellic, and meaty) in human-rating tasks. This is limiting the scope of the analyses, and it is not clear how these descriptors are chosen.

Questions for authors
  • (See weaknesses) Could the authors clarify the main motivation and the expected mechanism of improvement that they are testing for?

  • Standardized performance metrics for the score predictor are missing (only AUROC on floral descriptor is given on line 236). How does it compare to the widely used Principal Odor Map (POM) or Random Forest baselines on the training descriptors? What justifies not using an established predictor such as POM as the upstream model and augmenting it with the MiniLM embedding to allow for the novel descriptors?

  • Which 28 descriptors are withheld, and why? What are the exact criteria? Considering that there is a strong co-occurrence structure between descriptors (e.g., jasmine is a subcategory of floral), can the authors quantify the correlation between the withheld and training descriptors?

  • Why is floral used as "the initial query"? What is the rationale behind this choice? In addition, there are 55 descriptors in POM data and 138 in GS-LF so the analysis should cover at least the main odor categories or a representative set of descriptors.

  • Can the authors clarify line 339? There is "floral/odorless share 74 molecules". Are there any molecules that have both odorless and floral descriptors? This is plausible in the datasets, but it is unrealistic in the real-world condition, and it can signal potential label noise.

  • Evaluation of Qwen3-8B only on a subset of molecules (line 194) makes the comparison to the other models difficult. Could the authors evaluate Qwen3-8B on the same tasks as Qwen3-4B and Phi-4-mini?

  • The change of prompt (LF vs FF) puts together several changes at once, which limits the analysis. Can the authors provide a systematic prompt interpolation between LF and FF variants?

Reviewer_jwZoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an audit in the LLMs interfaces in the olfaction domain, that get scores from a frozen olfactory predictor. The idea is that the LLM is choosing between two molecules for an odor descriptor, and its choices are compared with a simple rule that picks the larger displayed score. The audit separates behavioral response, assignment advantage, retention of rule-correct decisions, and added value over the rule. It evaluates small open models (Qwen3 1.7B/4B/8B, Phi-4-mini) and one hosted Qwen3-32B on released odor annotations and human panel ratings.

Critical strengths

The paper studies LLMs in olfaction, which is a very underexplored modality. Testing how LLMs use predictions from an odor model is a relevant question as such interfaces start to appear.

Although each comparison is standard (counterfactual evidence swaps, position/label controls, negative-flip-style retention, and risk-coverage at matched coverage), combining them to audit an LLM placed on top of a scientific predictor is sensible, and they are applied thoroughly.

Critical weaknesses

There is no new method or metric. Retention is close to the known negative-flip idea from model-regression work. The main contribution is organizing these existing comparisons into an audit framework, but the methodological novelty is limited.

Several analyses that are standard in related work are missing. Selective comparisons are made at a single coverage determined by the model's abstentions, rather than using full risk-coverage curves ranked by model confidence (e.g. token probabilities). Prompt sensitivity is tested with only one revision, although that revision changed retention substantially (1/14 to 14/14 pairs on clear floral). Voting is not compared with standard position-debiasing methods such as PriDe (Zheng et al., 2024) or contextual calibration (Zhao et al., 2021).

The results are mostly null: no model demonstrates added value over the score rule, and the best case is reproducing it. This is expected and partly built into the setup. Thinking is disabled, output is capped at 64 tokens, only the first line is parsed, and the prompt tells the model not to output an explanation (Appendix F). With no room to reason, the models mostly follow the larger score, even in conditions where SMILES and dispersion are also shown. The setup therefore does not test whether more capable or reasoning-enabled models could extract more value from the evidence.

The main retention finding raises concerns about position bias. All 24 unstable pairs choose position A twice (Table 2), and in the supplementary material, the 11 pairs LF loses have smaller score gaps. The difference is real (11 vs. 0 discordant pairs), but the observed pattern is consistent with position-dependent selection and does not establish that the effect reflects a new or distinct failure mode.

The models are small and narrow. All are 1.7B-8B plus one 32B, with no frontier or reasoning-enabled models.

The predictor is custom (Morgan fingerprints + MiniLM) rather than an established one such as POM/OpenPOM, which makes the results harder to compare with other olfaction work. A custom predictor is fine for a controlled audit, but the results should also be shown with an established one to assess generalizability.

The samples are very small. The main released tasks have 101 pairs (nutty has only 17), and there are only 18 clear floral human pairs. In the 32B clear-pair risk comparison, the model answers 22 floral, 5 roasted, 6 meaty, and 0 caramellic presentations. The results for roasted and meaty are particularly sparse, and the absence of errors in these small samples does not establish equivalence or reliable risk guarantees.

Most of the study is post hoc and exploratory. Follow-up protocols were frozen after earlier results were inspected, and the wording revision was motivated by observed behavior. An independently sampled, pre-specified evaluation is needed to show whether the reported patterns persist beyond the cases that motivated the analyses.

Intervals are unadjusted for multiple comparisons.

There is no learned baseline on the same evidence. Without one, it is difficult to distinguish whether the lack of added value reflects limitations of the LLMs or the limited information in the evidence. More broadly, the experiments do not establish whether the observed limitations arise from the evidence, the interface, or the model's capabilities.

The paper is hard to follow, starting with the abstract and introduction. The abstract does not clearly state the task (choosing between two molecules given predictor scores). It lists raw counts (158/202, 67/78, 18/63) before the reader knows what they refer to, and it does not summarize the main finding in plain terms: the LLMs at best reproduce the score rule and do not demonstrate added value beyond it.

The introduction gives only one sentence of motivation for placing an LLM on top of an odor predictor. It also uses terms like "retention opportunities", "correction opportunities", "evidence reassignment" and "supplied-target reading" before they are defined. The rest of the text is very dense.

Questions for authors
  1. Can you add a learned baseline on the same evidence, such as a logistic regression on means, dispersion and SMILES? If it also can’t beat the rule, “no added value” reflects the evidence rather than the LLM.

  2. Why disable thinking, cap output at 64 tokens and forbid explanations? Would reasoning-mode or frontier models change the conclusion?

  3. How much of the retention difference remains after controlling for position bias, for example by debiasing the A/B choice or analyzing only pairs with larger score gaps?

  4. Consider adding POM/OpenPOM predictors too.

  5. Can the follow-up findings be reproduced on pairs not used to design them? For example, the revised prompt and the 32B model could be run, with the protocol fixed in advance, on the 20 human pairs per query not used in the follow-ups, and on new pairs from other withheld descriptors in OdorNet.

  6. Could you rank answers by model confidence (e.g. token probabilities) and report full risk-coverage curves or AURC?

  7. How sensitive are the conclusions to other paraphrases of the prompt?

  8. How does voting compare with standard position-debiasing methods such as PriDe or contextual calibration?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.