- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces RxnHaystack, a benchmark with 100 questions over 122,456 USPTO reactions. It tests how well models analyze large structured datasets. The tasks cover exact retrieval, property filtering, reaction classification, and multi-step route reasoning. The authors compare plain prompting, CodeAct, and RLM. They also fix answer counts or provide correct chemistry rules to study errors. Tools improve retrieval and deterministic calculations, but models still struggle with chemistry judgments and complex routes when the full dataset is available. The main contribution is the benchmark and the tests used to study these failures.
Critical strengths
- The benchmark covers several operations within the same dataset. This makes task comparisons useful. Reference answers are generated by code, reducing reliance on LLM judges.
- Tests with a fixed answer count or a supplied chemistry rule help check where errors come from. These controls add value beyond model rankings.
- The paper evaluates five models and reports task-level results, running costs, and execution traces. These details help explain errors behind the average scores.
Critical weaknesses
-
The benchmark has only 100 fixed questions. Some are close variants, such as six thresholds for the same molecular-weight filter. Repeated runs do not add task diversity. All tasks also come from one USPTO dataset. There is limited evidence that the findings hold for new questions or other scientific data.
-
Several tested models are mini, Flash, or Haiku versions, and full-corpus RLM runs use low reasoning effort. Low scores under these settings do not establish how well stronger systems could perform. A comparison with stronger models and higher reasoning effort would make the difficulty claims more convincing.
-
The link to real research needs is not clear enough. Lookup and property filters are standard data operations. Harder tasks, such as recovering routes with a hidden final step, need clearer use cases. The paper should explain why researchers need these queries and whether solving them improves a real analysis workflow.
-
Only RLM can access the full dataset. A code agent with the same data and RDKit, but no submodel calls, is missing. This comparison changes both data access and reasoning. It does not show what recursive calls add. A baseline with the same data access is needed.
-
The valid answer set for truncated-route tasks is unclear. Models must recover routes while the final step is hidden. Appendix A.8 tests extra target information, but does not explain how valid alternatives outside the stored routes are scored. Until this is clear, low F1 cannot be fully attributed to poor chemistry reasoning.
Questions for authors
- Can the authors add new targets and reaction templates, and check whether the main findings hold on another reaction dataset?
- Can the authors test stronger models and higher reasoning effort, and compare RLM with a no-subcall code agent using the same data, tools, and budget?
- Can the authors provide a real research use case and explain how other valid routes are judged and scored in the truncated-route tasks?
Thank you for your careful, comprehensive, and detailed response. Your rebuttal has persuaded me to reconsider some of my initial views. On my first reading, I misunderstood the paper’s research objectives, which led to some unfair assessments. I now understand that the paper primarily investigates how models and agent systems process increasingly large databases and what causes their failures. I suggest avoiding the term “long-context,” as it can be interpreted as referring to the model’s own context window rather than the scale of the underlying database. This clarification has led me to revise my overall assessment.
Nevertheless, I believe the paper still has substantial room for improvement. For a paper introducing a new benchmark, the experimental comparisons and analyses across models and agent frameworks remain relatively limited. The categorization of the capabilities underlying different failures is also rather coarse, and the findings offer limited novelty or insight.
Taking these considerations into account, I am raising my final overall score to 2.