acceptodds
Under review as a conference paper at ICLR 2027

RxnHaystack: Beyond Needles in Long-Context Structured Scientific Data

Abstract

Large language models (LLMs) deployed in AI-for-science increasingly confront structured datasets that far exceed their context windows. Existing long-context benchmarks, however, rarely distinguish failures of dataset-scale access, systematic execution over structured objects, and domain-specific reasoning. We introduce RxnHaystack, a diagnostic benchmark over 122,456 cleaned USPTO reactions. Its 100 questions span exact retrieval, deterministic aggregation, reaction-level inference, and relational reasoning over synthesis graphs. We evaluate plain prompting, CodeAct, and Recursive Language Models (RLMs) across 21,500 question-level trajectories. Plain-prompting performance declines in every tier as the supplied context grows from 100 to 500 reactions. CodeAct remains stable on Tiers 1–2 but declines on the harder Tiers 3–4 as contexts extend to 1,000 reactions. Both interfaces are bounded by the model context window, preventing them from receiving very large corpora in full. Recently proposed RLMs address this access constraint through recursive processing beyond the prompt window. Their full-corpus setting exposes agents to all 122,456 reactions, enabling more demanding corpus-wide questions and revealing which challenges remain once the complete dataset is accessible. Across Qwen3.5-397B-A17B, DeepSeek V4 Flash, Gemini 3.7 Flash, GPT-5 mini, and Claude Haiku 4.5, mechanically specified graph operations average 0.81 macro-F1, while mechanism classification averages 0.34 and route and multi-constraint tasks average 0.08. In controlled tests, chemistry-intensive performance falls as more reactions must be searched, even when every corpus contains one correct reaction. Providing the correct chemistry rule, but not the answers, improves classification, yet models still fail to apply it consistently across the entire corpus. Together with deterministic ground truth and inspectable traces, these controls turn aggregate performance into a diagnosis of why an agent fails, providing a reproducible testbed for agents that must operate systematically over large structured scientific data.

Primary area:
datasets and benchmarks
Keywords:
Long-context Reasoning, AI for Science, AI for Chemistry, Structured Data

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SsR9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces RxnHaystack, a benchmark with 100 questions over 122,456 USPTO reactions. It tests how well models analyze large structured datasets. The tasks cover exact retrieval, property filtering, reaction classification, and multi-step route reasoning. The authors compare plain prompting, CodeAct, and RLM. They also fix answer counts or provide correct chemistry rules to study errors. Tools improve retrieval and deterministic calculations, but models still struggle with chemistry judgments and complex routes when the full dataset is available. The main contribution is the benchmark and the tests used to study these failures.

Critical strengths
  1. The benchmark covers several operations within the same dataset. This makes task comparisons useful. Reference answers are generated by code, reducing reliance on LLM judges.
  2. Tests with a fixed answer count or a supplied chemistry rule help check where errors come from. These controls add value beyond model rankings.
  3. The paper evaluates five models and reports task-level results, running costs, and execution traces. These details help explain errors behind the average scores.
Critical weaknesses
  1. The benchmark has only 100 fixed questions. Some are close variants, such as six thresholds for the same molecular-weight filter. Repeated runs do not add task diversity. All tasks also come from one USPTO dataset. There is limited evidence that the findings hold for new questions or other scientific data.

  2. Several tested models are mini, Flash, or Haiku versions, and full-corpus RLM runs use low reasoning effort. Low scores under these settings do not establish how well stronger systems could perform. A comparison with stronger models and higher reasoning effort would make the difficulty claims more convincing.

  3. The link to real research needs is not clear enough. Lookup and property filters are standard data operations. Harder tasks, such as recovering routes with a hidden final step, need clearer use cases. The paper should explain why researchers need these queries and whether solving them improves a real analysis workflow.

  4. Only RLM can access the full dataset. A code agent with the same data and RDKit, but no submodel calls, is missing. This comparison changes both data access and reasoning. It does not show what recursive calls add. A baseline with the same data access is needed.

  5. The valid answer set for truncated-route tasks is unclear. Models must recover routes while the final step is hidden. Appendix A.8 tests extra target information, but does not explain how valid alternatives outside the stored routes are scored. Until this is clear, low F1 cannot be fully attributed to poor chemistry reasoning.

Questions for authors
  1. Can the authors add new targets and reaction templates, and check whether the main findings hold on another reaction dataset?
  2. Can the authors test stronger models and higher reasoning effort, and compare RLM with a no-subcall code agent using the same data, tools, and budget?
  3. Can the authors provide a real research use case and explain how other valid routes are judged and scored in the truncated-route tasks?
Reviewer_a1FYOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
1
Read review
Paper summary

This paper introduces a reaction-based benchmark for evaluating chemical capabilities, with tasks that vary along two dimensions—operation demand and chemistry-reasoning demand—and presents exploratory experiments.

Critical strengths
  1. The paper constructs a novel dataset. The distinction between operation demand and chemistry-reasoning demand is interesting.
  2. The categorization of failure mechanisms in chemistry tasks is insightful.
Critical weaknesses

The paper lacks a clear focus, the experiments and analysis are insufficient, and the study does not yield any meaningful conclusions.

The application scenarios described in the paper fall more naturally within the scope of agent-based systems, where one would not expect an LLM to complete these tasks solely through in-context processing. The experiments primarily compare three relatively simple agent frameworks and report performance aggregated across different models within each framework. However, the paper does not clearly articulate this framing. Instead, the introduction emphasizes limitations of LLMs’ long context windows, seemingly suggesting that the entire reaction corpus should be placed within the model’s context window. The absence of a full-corpus CodeAct setting appears to further support this interpretation. The authors therefore seem unclear about the research question they intend to investigate.

Furthermore, the benchmark is extremely small, containing only 100 questions, even though such questions are not difficult to generate. The experimental evaluation is also very limited: it does not adequately evaluate different agent frameworks and environments, considers only three relatively basic environments, and omits full-corpus results for CodeAct. The analysis is likewise insufficient and does not yield meaningful or insightful conclusions.

Questions for authors

None.

4 comments

Thank you for your careful, comprehensive, and detailed response. Your rebuttal has persuaded me to reconsider some of my initial views. On my first reading, I misunderstood the paper’s research objectives, which led to some unfair assessments. I now understand that the paper primarily investigates how models and agent systems process increasingly large databases and what causes their failures. I suggest avoiding the term “long-context,” as it can be interpreted as referring to the model’s own context window rather than the scale of the underlying database. This clarification has led me to revise my overall assessment.

Nevertheless, I believe the paper still has substantial room for improvement. For a paper introducing a new benchmark, the experimental comparisons and analyses across models and agent frameworks remain relatively limited. The categorization of the capabilities underlying different failures is also rather coarse, and the findings offer limited novelty or insight.

Taking these considerations into account, I am raising my final overall score to 2.

AuthorsOpenReview ↗

Final response to reviewer a1FY (Part 1/3)

Thank you for taking the time to go back through the paper and our rebuttal, and for revising your assessment. We appreciate your fairness in reconsidering your initial reading. We can see how “long-context” suggested that we meant to place the whole database in the model’s prompt. In the revision we will use “corpus-scale” for the size of the collection, reserve “context window” for the input the model itself sees, and change the title accordingly.

We have also posted our final clarifications, visible to all reviewers. They set out the contribution, the findings relative to prior work, and the scale and cost of the evaluation. Below we address each of your remaining points directly, so that this reply can be read on its own.

On the comparisons across models and agent frameworks

The benchmark itself is the contribution. The interfaces show which difficulties persist as an agent gains code execution and access to a larger collection, and the benchmark does not favor any of them. Our evaluation covers seven models and four inference settings:

  • Main study. Five models from five providers (two open-weight, three proprietary) under plain prompting, CodeAct, and RLM, at several corpus sizes, with five repetitions per configuration. That is 21,500 question-level trajectories.
  • Rebuttal. Two frontier models, GPT-5 and Gemini 3.1 Pro, under full-corpus CodeAct, RLM, and RLM without sub-calls, at two reasoning-effort levels, with the same corpus, tools, and call and time limits (Tables R5–R6). We also ran cross-corpus checks on USPTO-480K and Rhea (Tables R3–R4).

The full-corpus CodeAct baseline you suggested sharpened the paper. Both harnesses received the same 122,456 records, the same Python/RDKit tools, and the same limits. Full-corpus CodeAct scored 0.583 and 0.322 F1 for GPT-5 and Gemini 3.1 Pro, compared with 0.929 and 0.640 for RLM with sub-calls disabled (Table R5). The gap therefore persists when the harnesses see the same data, which the original comparison could not establish.

This breadth matches or exceeds published agent benchmarks, where every additional framework multiplies the cost of every model:

BenchmarkVenueModelsFrameworks / inference settings
RxnHaystack (ours)This submission5 + 2Plain prompting, CodeAct, RLM, and RLM without sub-calls; several corpus sizes; 5 repetitions
ScienceAgentBenchICLR 202553: direct prompting, OpenHands CodeAct, self-debug
MLE-benchICLR 202543 scaffolds, all three on one model; four models on one scaffold
DiscoveryBenchICLR 202533: CodeGen, ReAct, DataVoyager
Tooling or Not Tooling?Findings of NAACL 202523: direct prompting, ChemCrow, ChemAgent
CORE-BenchTMLR 202522 agents, one run only once because of cost
ChemCrowNature Machine Intelligence 202412: with and without tools
Recursive Language ModelsarXiv 202526 configurations
RULER / OOLONG / LongProcCOLM 2024 / 2026 / 202517 / 20 / 231 shared prompting protocol

Benchmarks that report many more models, such as RULER, OOLONG, and LongProc, evaluate them under a single prompting protocol. That costs far less per model than running agent harnesses over a large corpus.

(Continued in Part 2/3.)

AuthorsOpenReview ↗

Final response to reviewer a1FY (Part 2/3)

On the granularity of the failure analysis

We agree that telling a developer their agent struggled with “reasoning” gives them little to go on. The paper therefore separates failures at two levels.

Individual traces. Appendix C assigns a failure mechanism only when the generated code, tool observations, or returned answer give direct evidence. It distinguishes structured execution, chemical abstraction, relational operations, and target or route reasoning (Appendix C, Table 7). Three examples:

  • One agent substitutes a literal reagent name for the structural change the scientist asked for.
  • Another makes correct intermediate decisions and then loses them when assembling its final answer.
  • A third links unrelated reactions through a common reagent.

These call for different fixes, even when the runs miss the same records. Figure 32 classifies outcomes across 20,771 trajectories and separates omissions from extra selections.

Controlled experiments. Three controls turn these categories into measured quantities:

  • Formulating a criterion vs. applying it. We supply the executable per-record test and leave the agent to find every match. This raises F1 by 0.39–0.59 on amide and mechanism questions and closes more than half of the gap to exhaustive execution for both tested models. On queries that ask for every qualifying seven-reaction chain, the same supplied tests help neither model, although exhaustive execution scores 1.000 (Table 5).
  • Corpus size vs. number of answers. With exactly one correct reaction at every corpus size, classification F1 falls from 0.671 to 0.212 for GPT-5 mini and from 0.630 to 0.187 for Qwen3.5. Most of the loss is precision (0.634 → 0.213), while property filtering stays at 0.96–1.00 (Figure 3, Table 4).
  • Operation complexity vs. the criterion. Explicit graph operations average 0.806 F1 and mechanism classification 0.336, and every model shows this ordering (Table 2).

In the revision, we will also report omissions and extra matches separately under each control.

On novelty and insight

Our common final clarification sets these out in full. In brief, three findings go beyond prior work:

  1. With code access, the bottleneck moves from aggregation to the per-record domain criterion. OOLONG gives models the correct label for every record. Scores rise by only 0.8–10.9 points, and its authors conclude that finding and combining records is the bottleneck. With code access, our agents find and combine records well: 0.90, 0.91, and 0.81 F1 for lookup, property filters, and explicit graph operations on the full corpus. What recovers performance is supplying the per-record criterion.
  2. Moving the data out of the prompt removes the scale penalty only when the criterion is given. Criteria that an agent writes for itself can look accurate on a prompt-sized sample yet admit many near misses on the full collection. Benchmarks bounded by the context window cannot reveal this.
  3. Difficulty follows the inferred criterion, not the complexity class of the operation. The RLM paper orders tasks by how their work grows with input size. For tool-using agents, a graph traversal under a stated rule is far easier than a single linear scan that requires a chemical judgment. This holds in every model, and again on 20 new questions over two patent collections (Table R3: 1.000 vs. 0.600–0.800).

The conclusion that remains is this. For agents answering queries over collections larger than their prompt, retrieving records and executing explicitly specified operations are largely handled. What remains hard is turning a domain question into an exact criterion and applying it exhaustively, and earlier evaluations could not expose that end-to-end.

(Continued in Part 3/3.)

AuthorsOpenReview ↗

Final response to reviewer a1FY (Part 3/3)

On the size of the benchmark

The 100 questions are a deliberate balance between diversity, statistical reliability, and cost. They span four tiers and 30 task configurations. For every model, its lowest-scoring explicit-graph family still outperforms its highest-scoring mechanism family, so the contrasts do not rest on a few questions.

Developing and evaluating the benchmark has cost about \7,000sofar:about7,000 so far: about \\5,000 in API charges and \2,000foropen−weightmodels.Repeatingourcoregridwithfivecurrentfrontiermodelswouldcostabout2,000 for open-weight models. Repeating our core grid with five current frontier models would cost about \\17,000, and about \34,000at200questions.At100questions,alabcanrunanewharnessonthefullcorpusforunder34,000 at 200 questions. At 100 questions, a lab can run a new harness on the full corpus for under \\350 per model per pass. Our common clarification gives the full breakdown.

Although the discussion is closing, we would still be very grateful if you could concretely convey to the AC (for their final meta-review) your remaining concerns on additional specific models, frameworks, or analyses that you would have liked to see, as this would directly shape our revision.

Thank you again for engaging with our work and helping us improve it.

Reviewer_KwEZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RxnHaystack, a benchmark intended to diagnose why language-model agents struggle with large structured scientific datasets. Built from 122,456 cleaned USPTO reactions, it contains 100 questions across structural retrieval, property aggregation, reaction classification, and synthesis-graph reasoning. Its central methodological contribution is separating dataset access, systematic execution, and chemistry-specific reasoning through deterministic ground truth, controlled corpus sizes, and inspectable agent traces. The authors evaluate five models using direct prompting, CodeAct, and recursive language models over 21,500 trajectories, supplemented by a limited tool-assisted human baseline. Code-based methods perform well on lookup and descriptor tasks, whereas mechanism classification and multistep route reasoning remain substantially weaker despite full-corpus access. Fixed-answer-count experiments and supplied-rule controls suggest that failures reflect both incorrect chemical rules and inconsistent execution. Although the diagnostics are informative, validation remains concentrated in one domain, with engineered answer-containing contexts and limited human coverage, leaving broader generalizability and benchmark realism insufficiently established.

Critical strengths

A key strength is the benchmark’s effort to disentangle failure modes that standard long-context evaluations often conflate. Its 100 questions over 122,456 reactions combine deterministic RDKit-based ground truth with controlled variations in corpus size, operation demand, and chemistry reasoning. The fixed-positive-count and rule-supplied experiments are particularly valuable because they distinguish access limitations from failures to formulate or consistently execute chemical rules. This makes the benchmark more useful diagnostically than a single aggregate score.

A second strength is the substantial experimental coverage and trace-level analysis. Comparing five models under plain prompting, CodeAct, and recursive language models, with repeated trials and resource measurements, provides evidence that full-corpus accessibility is insufficient for reliable scientific reasoning. The gap between explicit graph traversal and multiconstraint route tasks is informative for future agent design. However, these contributions remain primarily diagnostic within one curated chemistry corpus, limiting their weight toward acceptance without broader validation.

Critical weaknesses

First, the benchmark construction limits how strongly its findings generalize to realistic scientific workflows. Every sampled context deliberately contains all scored positive reactions, effectively removing a substantial part of the retrieval challenge. Furthermore, synthesis graph relationships rely on exact molecular matching rather than experimentally validated chemical feasibility. Although these choices enable controlled evaluation, they simplify practical scientific reasoning and potentially limit the ecological validity of the conclusions.

Second, the experimental validation does not fully establish the broader claims. All tasks originate from a single USPTO reaction corpus, making transferability to other structured scientific domains uncertain. The human baseline covers only 43 of 100 questions with three volunteers, providing limited calibration of task difficulty. Additionally, differences in reasoning settings and corpus accessibility across evaluation paradigms complicate direct comparisons. While the diagnostic insights are valuable, these limitations weaken the evidence that the observed failure patterns are broadly representative.

Questions for authors

see weakness

Comments on the paper

5 comments
Area_Chair_V91EOpenReview ↗

Reminder to Reviewers: Please Engage in the Discussion

Dear Reviewers,

As already mentioned in my message to the authors, I would kindly ask you to read their responses and engage in the discussion.

Thank you!

Best regards, AC

AuthorsOpenReview ↗

Final clarification on contribution, novelty, and scale of the benchmark (Part 1/4)

Dear Area Chair and reviewers,

We sincerely thank the Area Chair for their assistance with the discussion and their thorough engagement, and we thank the reviewers for engaging with our work and reading our rebuttals. As the discussion period is closing, we would like to highlight clarifications on the main points around our contribution, novelty, relevance to the ICLR community, and the scale of the benchmark and its analyses. We thank you in advance for considering this rather lengthy response, which is our final opportunity to clarify our contributions and claims.

Contribution. First and foremost, our contribution is introducing an end-to-end benchmark grounded in real scientific workflows, for evaluating AI agents meant to assist scientists. Solving the benchmark integrates demands that existing evaluations mostly test in isolation a) retrieval in corpora too large to fit in the models’ context windows b) reasoning at graded levels of difficulty, and c) executions at graded levels of algorithmic complexity, from linear scans to pairwise joins and graph traversal across the massive corpora. Crucially, not only each bottleneck is controllable, but also the answers are exactly verifiable (no free-form text or LLM judgement), and every question is accompanied by its ground-truth answer set and the code that computes it. Together, these properties make RxnHaystack a highly controlled stress test for any new agent harness intended to support scientific discovery. We believe this is directly relevant to the part of the ICLR community working on AI agents, AI co-scientists, and AI for science.

Why Chemistry. Building the benchmark on chemistry does not narrow its relevance, rather, it enabled us to capture what domain experts would actually find useful, while the demands being evaluated (a-c above) are quite general; Many scientific analyses require searching collections larger than what fits the context of an LLM, from perturbation screens in biology to materials databases (both cited in our introduction). Translating a domain criterion into an exact rule is a step most scientific workflows share; chemistry is the domain where we have the expertise to define and verify those rules. The execution axis is defined by algorithmic complexity alone, not by chemistry (§3.2). We therefore see the goal and substance of RxnHaystack as relevant to the ICLR community in its own right, and as an invitation for similar benchmarks grounded in the workflows of other scientific fields, so that agents for scientific discovery are assessed on tasks experts need done. The findings below show what this design reveals that earlier evaluations did not.

On retrieval degradation and past work. Interestingly, we find that retrieval is not the bottleneck for the harnesses we consider. Retrieval degradation is real, but only for the plain LLM baseline that has to rely on its context, which we agree is well established (when the evidence is moved or reworded [1, 2], when distractors are added [3, 4], and even when the evidence remains recoverable [5]). But our Figure 2a,b show retrieval not to be a bottleneck for more elaborate harnesses, leaving the burden on reasoning and execution. The three findings below locate that difficulty and show where it departs from the closest prior work.

References

(Continued in Part 2/4.)

AuthorsOpenReview ↗

Final clarification on contribution, novelty, and scale of the benchmark (Part 2/4)

  1. With code access, the bottleneck moves from aggregation to the per-record domain criterion. In comparison to prior work, OOLONG [6] asks models to label each record in a long prompt and then aggregate. Giving models the correct label for every record improves scores by only 0.8–10.9 points, so the authors conclude that finding and combining records, not labeling them, is the bottleneck. We find the reverse for the agent harnesses we evaluate. Finding and combining records is largely handled: exact lookup, property filters, and graph operations with an explicit linking rule reach 0.90, 0.91, and 0.81 F1 on the full corpus. Agents fail instead on the per-record judgment, which requires chemistry; for example, an amide in the product may already have been present in a reactant. When we give the agent the executable test for this judgment, while it still has to find every match itself, F1 rises by 0.39–0.59 on amide-coupling and mechanism questions. That closes more than half of the gap to exhaustive execution in every model–family pair (e.g., 0.413 → 1.000 on amide couplings; Table 5). Unlike a full plan (SciTaRC [7]) or expert guidance (ScienceAgentBench [8]), this intervention supplies only the per-record test, so it separates formulating the criterion from applying it. A correct test is not always enough: on queries that require enumerating every qualifying seven-reaction chain, neither model improves, although exhaustive execution scores 1.000. Thus, the bottleneck moves from aggregation to the domain criterion, and for some queries to the search that applies it.
  2. Moving the data out of the prompt removes the scale penalty only when the criterion is given. To separate corpus size from the number of answers, we hold the number of correct reactions at one and grow the corpus the agent searches through code, from 100 records to the full collection. Property filtering stays at 0.96–1.00 for GPT-5 mini, while reaction classification falls from 0.671 to 0.212 for GPT-5 mini and from 0.630 to 0.187 for Qwen3.5 (Figure 3a,c). At a fixed 5,000 records, raising the number of correct reactions from one to twenty increases classification F1 (0.337 → 0.525 and 0.334 → 0.526; Figure 3b,d), so the loss is not explained by the number of answers. It is mostly lost precision: for GPT-5 mini, precision falls from 0.634 to 0.213 while recall falls from 0.800 to 0.684 (Table 4). Criteria the agent writes for itself look accurate on a few hundred records but admit many near misses across the full collection. An agent validated on a prompt-sized sample can therefore return many extra matches when a scientist runs it on the whole collection, which benchmarks that stay within the context window cannot reveal.
  3. For tool-using agents, difficulty follows the inferred criterion, not the complexity class. The RLM paper [9] orders long-input tasks by how their work grows with input size (constant, linear, quadratic), and for base models the pairwise task is by far the hardest. In our benchmark, explicit graph operations, which traverse linked reactions, average 0.806 F1 across five models, while mechanism classification, a single linear scan, averages 0.336; every model shows this ordering (Table 2). The rebuttal experiments reproduce it on new questions: GPT-5 and Gemini 3.1 Pro reach 1.000 F1 on exact two-step graph search on both patent collections at full size, versus 0.600–0.800 on single-reaction motif classification (Table R3). With code, following a stated link is a standard algorithm the agent writes correctly, whereas classifying a mechanism requires the agent to work out the chemical test itself.

What conclusion remains. Taking prior work into account, we believe the conclusion that remains is this. For agents that answer queries over structured collections larger than their prompt, retrieving records and executing an explicitly specified operation are largely handled. What remains is turning a domain question into an exact criterion and applying it exhaustively. Benchmarks that keep the data in the prompt, or whose per-record labels are easy, cannot expose these failures. We also agree with the point that our experiments locate where performance breaks down rather than explain how agents retrieve evidence. We will frame the research question that way in the revised introduction, together with an expanded related-work section.

References

(Continued in Part 3/4.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.