acceptodds
Under review as a conference paper at ICLR 2027

SHELDON: Reasoning through Dual Search over Hypotheses and Observation Spaces

Abstract

Test-time scaling has enabled large language models to solve increasingly complex reasoning, discovery, and optimization tasks, but existing approaches often suffer from fixed search strategies, or hypothesis-centric reasoning that under-explores informative evidence. Inspired by the hypothetico-deductive method (Hempel, 1945) and dual-space models of scientific discovery, we introduce SHELDON, a scientific reasoning harness that jointly models hypothesis and observation spaces. By treating observations as first-class objects, SHELDON can identify gaps in its current understanding, uncover new hypotheses, and design investigations that better distinguish among alternatives. Across diagnosis, abstract reasoning, and optimization benchmarks, SHELDON outperforms prior state-of-the-art methods by upto 26.7 points. Moreover, on system root cause analysis tasks, SHELDON discovers 29.8% points more ground-truth causes and investigates 34.4% points more discovered causes than other baselines, demonstrating that joint hypothesis-observation search leads to broader and more effective scientific reasoning at test time.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
test-time search, scientific reasoning, agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7d1gOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes SHELDON, an LLM agent harness for investigative reasoning, built on the hypothetico-deductive method and the dual-space search model of Klahr & Dunbar (1988). The harness keeps explicit hypothesis and observation spaces, each divided into axes and regions, and links observations to hypotheses with categorical evidence relations (support, refute, neutral, unresolved). In each round the agent explores, analyzes each hypothesis, compares hypotheses with one another, and then keeps, repairs, refines, or shelves them, with a cooldown on shelved ones. Using GPT-5.6-terra, the authors evaluate on medical diagnosis (DiagnosisArena), incident root-cause analysis (a modified ORCA-Bench), ARC-AGI-2, QNN circuit topology, DOCKSTRING, and system-optimization tasks (ADRS). They report gains over the baselines, notably on DiagnosisArena (84.00 vs 78.00) and ORCA-Bench (66.67 vs 40.00). Ablations show drops of up to 14 points, and SHELDON covers more ground-truth causes than GoS.

Critical strengths
  1. Grounding the harness in Klahr & Dunbar's dual-space search is coherent, and generalizing the experiment space to an observation space (Sec. 4.1) is a reasonable adaptation for agentic settings.
  2. The task set covers problems with and without a scoring function (diagnosis, root-cause analysis, ARC, program optimization) in one harness.
  3. Reproducibility material is extensive (prompts, state structures, hyperparameters, task IDs, per-method cost), and Sec. 6 goes beyond headline scores with cause-coverage analysis and trajectory case studies.
Critical weaknesses
  1. It is unclear whether the gains come from the dual-space structure or from extra compute. SHELDON spends $2.91 per DiagnosisArena case and $6.90 per ORCA-Bench incident, while ReAct, Codex and RLM spend $0.02-$1.56 (Table 5). The comparison with GoS at similar cost helps, but no single-agent baseline gets the same budget through generic test-time scaling, such as best-of-N, self-consistency, or parallel rollouts with an aggregator like RSA. Without that, the paper's main claim, that explicit hypothesis/observation structure is what helps, is not isolated.
  2. The ORCA-Bench setup rewards breadth. The authors changed the task to ask for "all plausible causes" and dropped causes that started after the reporting time (App. D). The verifier counts an incident as solved when the report names every ground-truth cause, so a harness that keeps and reports many hypotheses has a natural advantage. The changes apply to all methods, but precision (non-ground-truth causes per report) is not given. The same concern applies to the 76.6% vs 46.8% coverage analysis.
  3. The contribution of each design element is not isolated. GoS, Arbor and ScientistOne already maintain hypothesis-evidence structures. SHELDON adds the axes/regions map, the comparative stage and the shelf/cooldown, but no ablation compares the axes/regions map with plain observation logging, and Table 3 does not say what "w/o observation space" removes. SHELDON also delegates implementation to GPT-5.6-luna, while the baselines use terra only. Two descriptions do not match the implementation: the PROVISIONAL attribute (Sec. 4.2) is missing from the Observation dataclass (App. E), and the QNN "human baseline" is never reported.
  4. The statistical evidence is thin. The DiagnosisArena (n = 50) and ORCA-Bench (n = 30) gaps are 3 and 8 cases. The tables appear to be single runs with no standard deviations. On ADRS, several tasks are ties or near-ties at saturation. The paper does not state the number of ARC-AGI-2 tasks or the scoring rule.
Questions for authors
  1. Can you compare against compute-matched generic scaling (best-of-N, self-consistency or RSA with ReAct/Codex) at about $2.9 and $6.9 per task?
  2. For which kinds of tasks should an explicit hypothesis/observation space help, and when might it hurt (e.g., by anchoring on early hypotheses)? Do the results across scored and unscored tasks support such a characterization?
  3. On ORCA-Bench, what is the precision of each method's report, and what are the results on the unmodified task?
  4. What does the axes/regions map add compared with a flat log of observations, and what exactly does "w/o observation space" remove? How much of the gain remains with a terra-only configuration?
  5. Is PROVISIONAL implemented, and what is the human baseline for QNN topology?
7 comments

We thank the reviewer for their insightful feedback. SHELDON's central contribution is actively finding new observations that lead to new and better-supported hypotheses, while rigorously testing candidate hypotheses against the evidence.

1. Generic Scaling and Computational Cost (W1, Q1)

Reviewer: It is unclear whether the gains come from the dual-space structure or from extra compute. Can you compare against compute-matched generic scaling

We expanded Codex’s computation through independent investigations and answer aggregation. Caching issues were resolved, and SHELDON's hyperparameters were revised for tasks lacking scoring functions. Four computation rounds were used. The previously reported four-round result now incurs an average cost \0.83perDiagnosisArenacaseand0.83 per DiagnosisArena case and \\2.21 per ORCA incident. We also report the round-3 ORCA checkpoint, averaging \$1.55 per incident. All comparisons use the revised configuration.

Rollout and aggregation protocol.

ORCA: Run three independent Codex rollouts. For aggregation, a separate Codex instance reviews the investigation trajectories and final reports from all three rollouts to produce the final answer. DiagnosisArena: Run five independent Codex investigations in parallel, prompting each toward a \0.15per−rolloutspendingtarget.Aggregateoutputsfromallfiveruns.Thefinalcostmaydifferfromtherealizedexpenditure:themeasuredcostforthescaledvariantsis0.15 per-rollout spending target. Aggregate outputs from all five runs. The final cost may differ from the realized expenditure: the measured cost for the scaled variants is \\0.65 per case, as reported in the table.

We evaluated three answer-combination strategies:

  • Majority voting/self-consistency: Majority voting on DiagnosisArena and consistency-based synthesis on ORCA.
  • Aggregation: A final Codex aggregator combines the independent investigations; on ORCA, it receives both their trajectories and final reports.
  • Best-of-N (oracle): Ground-truth evaluation selects the best candidate among the independent investigations, providing an evaluation-only reference. This is the upper bound of the methods.

Lower-Budget ORCA Comparison

Round 3 SHELDON at an average cost of \$1.55 per incident. Across three runs, it gets a 57.8% mean RCA, with mean reward 0.729. These are averages across evaluation runs. The scaled Codex baselines below, each use three independent rollouts per incident and were evaluated once.

MethodRCA (%)Cost per task ($)
Codex Consistency43.331.72
Codex Aggregation53.331.75
Codex best of n (Oracle)56.671.72
SHELDON, round 357.781.55

RCA (Root-Cause Accuracy) means if the agent found all the root cause analysis for a task.

At this operating point, SHELDON has 4.4 percentage points higher mean RCA accuracy at approximately 11% lower cost than Codex aggregationReward at round 3 is lower than both aggregation and oracle selection, so the result shows a accuracy–cost advantage. The scaled baselines have not been repeated.

Four-Round and DiagnosisArena Results

SHELDON's accuracy values in the following table, 88.00% on DiagnosisArena and 66.67% on ORCA, are the best of three runs. We report all three SHELDON runs and their means separately in the robustness section. The scaled Codex strategies were each evaluated once. The cost columns describe the reported per-run expenditure; selecting SHELDON's best-of-three score requires three evaluations.

MethodORCA RCA (%)ORCA RewardORCA $/taskDiagnosis AccuracyDiagnosis $/task
ReAct46.670.6220.4572.000.02
Codex50.000.6680.5776.000.04
RLM40.000.6450.3978.000.09
GoS23.330.4334.6068.002.24
SHELDON66.670.8212.2188.000.83

RCA means Root-Cause Accuracy – if the agent found all the root cause analysis for a task. Reward: Calculated using a rubric 0 - 3, where three means most complete answer, and 0 means completely hallucinated answer. The calculated reward for each task is then divided by 3.

The lower-budget comparison above is our direct response to the ORCA compute concern. The table here retains the four-round and DiagnosisArena results with their original reporting conventions. SHELDON's three-run means at four rounds are 61.1% on ORCA and 84.7% on DiagnosisArena; the 66.67% and 88.00% values in this table are best-of-three scores. The paired comparisons in the robustness section provide the available repeated-run evidence against the original single-agent baselines.

We will add these results to the next version of the paper.

2. ORCA Evaluation and Root-Cause Precision (W2, Q3)

Reviewer: The ORCA-Bench setup rewards breadth. On ORCA-Bench, what is the precision of each method's report, and what are the results on the unmodified task?

To clarify the setup: the original ORCA-Bench task already states that there may be multiple root causes, and its verifier already requires every gold cause to be named. We did not add causes to the gold set. We made two changes, applied identically to all methods. First, we added a line to the task description clarifying this requirement; the requirement itself is already part of the original task. Second, we dropped gold causes that began after the incident was reported, since an operator could not have observed them at that time; this only shrinks the gold set. We will make this wording precise in Appendix D, and we report results on the unmodified benchmark below.

ORCA-Bench Metric Definitions

We use the original definitions in ORCA-Bench, Appendix H. RCA accuracy requires every gold cause; hallucination rate flags a nonempty report with no correct cause. Precision is the fraction of extracted causes that are correct (zero for no attributions); recall is the fraction of gold causes recovered. Values are averaged over incidents. A flag without a variant matches any gold variant. Thus, high RCA accuracy and low hallucination rate can coexist with extra incorrect causes; precision addresses that concern.

Reward is normalized RCA depth: each incident's mean 0–3 rubric score divided by three, then averaged across incidents. Incorrect or misleading claims reduce rubric scores. This follows the original verifier's scoring and reward computation.

Original ORCA Setup

We also ran the unmodified benchmark, with the original instruction and all gold causes, for SHELDON and the strongest baseline, Codex. Both methods score lower than in our setup (Codex 26.67% vs. 50.00%; SHELDON 33.33% vs. 66.67%), consistent with the issues above affecting all methods alike. SHELDON still leads on RCA accuracy (33.33% vs. 26.67%) and on cause-level precision (93% vs. 77%).

MethodRCA (%)RewardHallucination rateGold causes credited (%)Precision (cause level)Recall (cause level)Cost per task
Codex26.670.56410.0041.177%43%0.61
SHELDON33.330.5703.3351.693%49%1.90

Four-Round ORCA Metrics

Under our modified task setup, the following four-round table reports 87% cause-level precision for SHELDON and its best-of-three RCA accuracy of 66.67%. These precision and hallucination-rate values are not measurements of the lower-budget round-3 checkpoint. Reporting precision together with recall and reward addresses whether higher coverage is accompanied by additional incorrect attributions. The separate robustness section reports the three-run comparison with the strongest original single-agent baseline.

MethodRCA (%)RewardHallucination rateGold causes credited (%)Precision (cause level)Recall (cause level)Cost per task
ReAct46.670.6226.6756.882%56%0.45
Codex50.000.66810.0058.071%59%0.57
Codex Consistency43.330.6676.6761.776%64%1.72
Codex best of n (Oracle)56.670.7703.3374.179%73%1.72
Codex synthesis53.330.7913.3370.473%73%1.75
RLM40.000.64510.0055.663%56%0.39
GoS23.330.43336.6725.953%29%4.60
SHELDON66.670.8213.3380.287%80%2.21

We will add these results to our next draft.

3. Isolating the Observation-Space Contribution (W3, Q4)

Reviewer: The contribution of each design element is not isolated.

Response:

Axes and regions are one supporting component; the main contribution is the process of discovering observations, generating hypotheses from them, and rigorously testing those hypotheses. SHELDON searches for observations without requiring an existing hypothesis to motivate the investigation. Unexpected or unexplained observations can therefore suggest new mechanisms and improve the hypotheses under consideration. Candidate analysis investigates each explanation, and comparative analysis seeks evidence that distinguishes competing explanations. The accumulated evidence determines which hypotheses are retained, revised, or discarded.

We conducted targeted ablations in which all settings remained unchanged except for the intervention specified in each ablation. The base model, task set, hypothesis-diversity instructions, candidate count, number of rounds, tool availability, and budget settings were held constant unless explicitly modified by the intervention. Actual expenditure was also comparable: **approximately \2.10perORCAincidentand2.10 per ORCA incident and \\0.80–0.85 per DiagnosisArena case across all ablation variants. Thus, each comparison isolated the effect of a single mechanism while preserving the rest of the system configuration.

  • Disable new-observation search: Generate hypotheses using only the task description, without consulting the case material.
  • Remove new observation instruction: Omit the prompt that instructs the system to prioritize newer observations.
  • Attach observations directly to hypotheses: Keep observations in context alongside the hypotheses.

We observe that relative to the full-system of SHELDON, attaching observations directly to hypotheses gives the largest differences: −18.67 points on DiagnosisArena and −17.78 points on ORCA. Disabling independent observation search gives differences of −16.67 and −11.11 points, respectively. Removing the observation-seeking instruction gives differences of −10.67 and −11.11 points. This shows that all the components of SHELDON help the system.

AblationDiagnosisArena (%)ORCA RCA (%)
SHELDON84.6761.11
Disable new-observation search68.00 (−16.67)50.00 (−11.11)
Remove new observation instruction74.00 (−10.67)50.00 (−11.11)
Attach observations directly to hypotheses66.00 (−18.67)43.33 (−17.78)

The full SHELDON row presents three-run means: 84.67% on DiagnosisArena and 61.11% on ORCA, using the three-round setup. Parentheses values indicate percentage-point differences from the full-system mean, calculated prior to rounding.

The observed pattern points to larger differences for independent observation search and evidence retention than for the axes/regions representation alone; these differences are descriptive, without uncertainty estimates for the ablation variants.

Our "Attach observations directly to hypotheses" ablations directly address the reviewer's question about structured-memory alternatives.

These results distinguish SHELDON from simply maintaining structured notes or refining hypotheses. The independent observation search allows the agent to investigate directions not motivated by existing hypotheses.

Importantly, the original (in the submission) observation-space ablation does not remove evidence access. Observations remain available within each round; hypotheses, associated evidence, test notes, and reasons for discarding candidates persist across rounds. Only the global observation ledger and explicit observation regions are removed.

Similarly, removing comparative analysis does not eliminate investigation: explore, candidate analysis, and exploit remain intact. The ablations therefore target specific mechanisms rather than removing the entire discovery process.

Model Allocation

For ORCA, Diagnosis Arena, ARC-AGI-2 and QNN, we use terra for all our experiments. Only ADRS uses Luna for delegated implementation within SHELDON, which is a weaker model than GPT-5.6-terra (used by the baselines). Thus, SHELDON's gains cannot be attributed simply to using a stronger implementation model. In fact, SHELDON achieves higher performance despite delegating implementation to a weaker model.

We agree that a terra-only SHELDON configuration would provide an additional useful control. We plan to investigate the effect of implementation-model choice in the next iteration and will clarify the model allocation and its limitations in the revised manuscript.

Reviewer_tiQ9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SHELDON, a harness for LLM agents that aims to improve the exploration and investigation of competing hypotheses. It organizes hypotheses and observations explicitly, using new observations to generate hypotheses and refine existing ones. Each round roughly follows four main steps:

  1. Explore: formulate hypotheses with distinct mechanisms.
  2. Analyze the candidate hypotheses: a separate tool-using agent gathers information, runs evaluations where available, and records evidence for each selected hypothesis.
  3. Compare multiple hypotheses: look for evidence that can distinguish between them.
  4. Adjust the hypotheses based on the evidence: retain, repair, refine, or set aside hypotheses, and carry the accumulated observations into the next round.

The framework is evaluated on diagnosis, system root-cause analysis, abstract reasoning, and optimization tasks.

Critical strengths

I find the interaction between the hypothesis and observation spaces quite interesting. The underlying dual-space idea comes from Klahr and Dunbar (1988), and adapting it to organize LLM-agent investigations is a worthwhile direction.

Critical weaknesses
  1. The gains are not adequately separated from differences in actual computation. On DiagnosisArena, SHELDON spends \2.91percase,comparedwith2.91 per case, compared with \\0.09 for RLM (Table 5). GoS spends a comparable amount and performs worse, so cost alone does not explain every result. However, the paper lacks a comparison that gives a simpler agent a similar actual budget through repeated independent investigations and answer aggregation, or candidate selection where an evaluator is available. A common budget cap does not ensure comparable computation.
  2. The experiments do not establish what the explicit observation space contributes beyond structured memory and the other search mechanisms. The “w/o observation space” variant is not defined clearly enough to tell which information and operations are removed. The harness also combines diversity instructions for hypothesis generation, parallel candidate investigations, repeated refinement, and combination or revival of previous hypotheses. These effects are not separated. The current ablations therefore do not establish that the gains specifically require the proposed observation-space organization rather than a simpler way of retaining and reusing evidence.
  3. The evaluation provides limited evidence of robustness. DiagnosisArena and ORCA-Bench use only 50 cases and 30 incidents, respectively. The six-point improvement over RLM on DiagnosisArena corresponds to three additional cases. Table 3 does not report repeat counts or uncertainty estimates, making it difficult to assess the stability of the ablation results.
  4. The ORCA result does not distinguish broader cause coverage from more selective reasoning. Its reported RCA accuracy requires covering all ground-truth causes but does not penalize additional incorrect causes (see ORCA-bench, Appendix H). Combining findings from multiple hypotheses may improve coverage under this metric. Without reporting precision or the number of incorrect causes included, it remains unclear whether the gain reflects better discrimination among explanations or simply broader inclusion of possible causes.
  5. The paper needs more careful proofreading. Examples include “upto” (line 20), “observatin” (line 122), “obserations” (lines 287–288), and “implementated” (line 289).
Questions for authors
  1. What exactly is removed in the “w/o observation space” ablation? In particular, are accumulated observations and their sources still available to later stages, and are actual model and tool-use costs comparable across variants?
  2. How does the explicit observation space compare with an agent that retains the same evidence in its context or structured notes? Could the authors provide a controlled comparison that keeps the hypothesis-diversity instructions, candidate count, investigation stages, and computational budget the same?
7 comments

We thank the reviewer for their insightful questions. Our new scaling the baselines results and controlled ablations provide evidence for the contribution of independently exploring and maintaining an explicit observation space.

1. Computational Budget and Stronger Baselines

Reviewer: The gains are not adequately separated from differences in actual computation.

We expanded Codex’s computation through independent investigations and answer aggregation. Caching issues were resolved, and SHELDON's hyperparameters were revised for tasks lacking scoring functions. Four computation rounds were used. The previously reported four-round result now incurs an average cost of \0.83perDiagnosisArenacaseand0.83 per DiagnosisArena case and \\2.21 per ORCA incident. To analyze performance with lower computational budgets, we also report the round-3 ORCA checkpoint, averaging \$1.55 per incident. All comparisons use the revised configuration.

Rollout and aggregation protocol.

ORCA: Run three independent Codex rollouts. For aggregation, a separate Codex instance reviews the investigation trajectories and final reports from all three rollouts to produce the final answer. DiagnosisArena: Run five independent Codex investigations in parallel, prompting each toward a \0.15per−rolloutspendingtarget.Aggregateoutputsfromallfiveruns.Thefinalcostmaydifferfromtherealizedexpenditure:themeasuredcostforthescaledvariantsis0.15 per-rollout spending target. Aggregate outputs from all five runs. The final cost may differ from the realized expenditure: the measured cost for the scaled variants is \\0.65 per case, as reported in the table.

We evaluated three answer-combination strategies:

  • Majority voting/self-consistency: Majority voting on DiagnosisArena and consistency-based synthesis on ORCA.
  • Aggregation: A final Codex aggregator combines the independent investigations; on ORCA, it receives both their trajectories and final reports.
  • Best-of-N (oracle): Ground-truth evaluation selects the best candidate among the independent investigations, providing an evaluation-only reference. This is the upper bound of the methods.

Lower-Budget ORCA Comparison

We additionally evaluate SHELDON at the round-3 checkpoint, at an average cost of $1.55 per incident. Across three runs, it gets a 57.8% mean RCA, with mean reward 0.729. These are averages across evaluation runs. The scaled Codex baselines below, each use three independent rollouts per incident and were evaluated once.

MethodRCA (%)$/ task
Codex Consistency43.331.72
Codex Aggregation53.331.75
Codex best of n (Oracle)56.671.72
SHELDON, round 357.781.55

RCA accuracy (Root-Cause Accuracy) means if the agent found all the root cause analysis for a task.

At this operating point, SHELDON has 4.4 percentage points higher mean RCA accuracy at approximately 11% lower cost than Codex aggregation. The difference from oracle-selected best-of-N is smaller: 1.1 percentage points at approximately 10% lower cost. The oracle uses ground-truth selection and establishes the upper bound for Codex. Reward at round 3 is lower than both aggregation and oracle selection, so the result shows a modest accuracy–cost advantage. The scaled baselines have not been repeated.

Four-Round and DiagnosisArena Results

MethodORCA RCA (%)ORCA RewardORCA $/taskDiagnosis AccuracyDiagnosis $/task
ReAct46.670.6220.4572.000.02
Codex50.000.6680.5776.000.04
Codex Consistency43.330.6671.7270.000.65
Codex Aggregation53.330.7911.7580.000.65
Codex best of n (Oracle)56.670.7701.7276.000.65
RLM40.000.6450.3978.000.09
GoS23.330.4334.6068.002.24
SHELDON66.670.8212.2188.000.83

SHELDON's accuracy values in the following table, 88.00% on DiagnosisArena and 66.67% on ORCA, are the best of three runs. We report all three SHELDON runs and their means separately in the robustness section. The scaled Codex strategies were each evaluated once. The cost columns describe the reported per-run expenditure; selecting SHELDON's best-of-three score requires three evaluations.

The paired comparisons in the robustness section provide the available repeated-run evidence against the original single-agent baselines.

We will add these results to the next version of the paper.

2. Contribution of the Observation Space

Reviewer: The experiments do not establish what the explicit observation space contributes beyond structured memory and other search mechanisms. The ablations do not isolate its effects.

Response:

We conducted targeted ablations in which all settings remained unchanged except for the intervention specified in each ablation. The base model, task set, hypothesis-diversity instructions, candidate count, number of rounds, tool availability, and budget settings were held constant unless explicitly modified by the intervention. Actual expenditure was also comparable: approximately $2.10 per ORCA incident and 0.80–0.85 per DiagnosisArena case across all ablation variants. Thus, each comparison isolated the effect of a single mechanism while preserving the rest of the system configuration.

  • Disable new-observation search: Generate hypotheses using only the task description, without consulting the case material.
  • Remove new observation instruction: Omit the prompt that instructs the system to prioritize newer observations.
  • Attach observations directly to hypotheses: Keep observations in context alongside the hypotheses.

We observe that relative to the full-system of SHELDON, attaching observations directly to hypotheses gives the largest differences: −18.67 points on DiagnosisArena and −17.78 points on ORCA. Disabling independent observation search gives differences of −16.67 and −11.11 points, respectively. Removing the observation-seeking instruction gives differences of −10.67 and −11.11 points. This shows that all the components of SHELDON help the system.

AblationDiagnosisArena (%)ORCA RCA (%)
SHELDON (three-run mean)84.6761.11
Disable new-observation search68.00 (−16.67)50.00 (−11.11)
Remove new observation instruction74.00 (−10.67)50.00 (−11.11)
Attach observations directly to hypotheses66.00 (−18.67)43.33 (−17.78)

The full SHELDON row presents three-run means: 84.67% on DiagnosisArena and 61.11% on ORCA, using the three-round setup. Parenthetical values indicate percentage-point differences from the full-system mean, calculated prior to rounding.

The observed pattern points to larger differences for independent observation search and evidence retention than for the axes/regions representation alone; these differences are descriptive, without uncertainty estimates for the ablation variants.

Our "Attach observations directly to hypotheses" ablations directly address the reviewer's question about structured-memory alternatives.

These results distinguish SHELDON from simply maintaining structured notes or refining hypotheses. The independent observation search allows the agent to investigate directions not motivated by existing hypotheses.

Importantly, the original (in the submission) observation-space ablation does not remove evidence access. Observations remain available within each round; hypotheses, associated evidence, test notes, and reasons for discarding candidates persist across rounds. Only the global observation ledger and explicit observation regions are removed.

Similarly, removing comparative analysis does not eliminate investigation: explore, candidate analysis, and exploit remain intact. The ablations therefore target specific mechanisms rather than removing the entire discovery process.

Why explicit instruction to look for new observations helps in exploration

We investigated why removing the explicit instruction to seek new observations negatively impacts SHELDON's performance. Our analysis shows that, despite access to the same set of tools, SHELDON makes significantly fewer tool calls to explore the case material when this instruction is omitted. This highlights simply providing tool availability and enabling iterative reasoning are not sufficient to ensure ongoing exploration of the case material/environment

On average per task, SHELDON used tools to discover new observations 10.8, 2.7, 1.4, and 1.2 times per round in Diagnosis Arena, and 8.2, 3.1, 2.3, and 2.0 times per round in ORCA-bench, whereas the variant without observation instruction did so 8.7, 0.2, 0, and 0 times in Diagnosis Arena, and 5.4, 0.3, 0.3, and 0 times in ORCA-bench.

3. Robustness and Repeated Runs

Reviewer: The evaluation provides limited evidence of robustness, particularly given the benchmark sizes and missing uncertainty estimates.

Response:

We conducted a targeted three-run robustness study for SHELDON and the strongest original single-agent baseline on each benchmark: Codex on ORCA and RLM on DiagnosisArena. We report every run and the mean for both methods in each paired comparison. The means below are reported separately from SHELDON's best-of-three headline scores.

We report an exact two-sided Wilcoxon signed-rank test: zero differences are discarded, tied absolute differences receive mid-ranks, and the p-value is computed from the exact permutation distribution of the signed-rank statistic over all sign assignments of the nonzero differences. We first average each case's performance over three runs, then pair the 30 ORCA or 50 DiagnosisArena case-level averages. Additionally, exact McNemar on a per-case majority vote is also statistically significant.

ORCA Bench (four-round SHELDON versus the original Codex baseline): Wilcoxon paired test p = 0.005

MethodRun 1Run 2Run 3Mean
Codex46.730.050.042.2
SHELDON60.056.766.761.1

Diagnosis Arena: Wilcoxon paired test p = 0.013

MethodRun 1Run 2Run 3Mean
RLM78747074.0
SHELDON88828484.7

We report the paired improvement in percentage points with a 95% confidence interval from a case-level bootstrap: 100,000 resamples of cases with replacement, each case keeping its three runs grouped, so repeated runs are never treated as independent. On ORCA-Bench, four-round SHELDON improves RCA accuracy over the original Codex baseline by +18.9 pp (95% CI [+10.0, +27.8]). On DiagnosisArena, it improves over RLM by +10.7 pp (95% CI [+3.3, +18.7]).

The ORCA p-value and confidence interval above apply to the four-round comparison with the original Codex baseline.

Repeated-Run Comparison with CORAL

The repeated-run results below indicate that SHELDON achieves higher mean performance than CORAL on all three reported ADRS tasks. Notably, SHELDON also outperforms CORAL on four out of six molecular optimization tasks, identifying three best-in-library molecules. We want to emphasize the importance of the consistency of improvements that SHELDON shows, rather than solely their absolute score, especially for tasks nearing saturation. Additionally, for optimization tasks, even small performance gains can be highly significant.

Our analysis of repeated runs further highlights the consistency of SHELDON:

On Cloudcast, CORAL achieved a mean score of 619.00 (± 0.000), while SHELDON achieved 618.09 (± 0.0173); on LLMSQL, CORAL scored 0.7115 (± 0.004) compared to SHELDON’s 0.7204 (± 0.003); and on Transaction Scheduling, CORAL obtained a mean score of 4077 (± 107.39) versus SHELDON’s 4198 (± 65.82).

For each method, we performed three independent runs to evaluate consistency. On ADRS tasks, SHELDON consistently achieves higher mean performance than CORAL, with minimal run-to-run variability. These improvements are particularly stable on Cloudcast and LLMSQL, whereas Transaction Scheduling exhibits higher variability.

We will clarify the number of repeated runs and report available uncertainty estimates .

4. ORCA Evaluation and Root-Cause Coverage

Reviewer: ORCA's evaluation rewards covering all ground-truth causes without penalizing additional incorrect causes. The gains may reflect broader inclusion

Response:

SHELDON's higher RCA accuracy does not come from naming more causes. In the four-round results below, it has the highest cause-level precision of all methods (87%; the next best is ReAct at 82%) and the joint-lowest hallucination rate (3.33%), while also recovering the largest share of gold causes (80% recall). Its reports therefore contain a smaller share of incorrect causes than any baseline's, so the gain reflects better discrimination among explanations rather than broader inclusion.

ORCA-Bench Metric Definitions

We use the original definitions in ORCA-bench. RCA accuracy requires every gold cause; hallucination rate flags a nonempty report with no correct cause. Precision is the fraction of extracted causes that are correct (zero for no attributions); recall is the fraction of gold causes recovered. Values are averaged over incidents. A flag without a variant matches any gold variant. Thus, high RCA accuracy and low hallucination rate can coexist with extra incorrect causes; precision addresses that concern.

Reward is normalized RCA depth: each incident's mean 0–3 rubric score divided by three, then averaged across incidents. Incorrect or misleading claims reduce rubric scores. This follows the original verifier's scoring and reward computation.

Reviewer_vCqmOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SHELDON (I like the name), a scientific reasoning harness that maintains a hypothesis and observation space. It alternates between exploration, candidate analysis, comparative analysis, and exploitation. Experiments cover an extensive set of tasks and baseline methods, with additional component ablations.

Critical strengths
  • The evaluation is very much extensive, covering a wide set of tasks, and shows promising results on several tasks. I appreciate the paper's attempt for broad task and baseline coverage.
  • For method construction, the reasoning process is reasonable and clearly organized. Explicit hypothesis, observations, and evidence relations make the investigation process easy to follow. The provided figures also seem carefully curated and help with understanding.
Critical weaknesses
  • The novelty of dual-space search needs clearer grounding. Iterating between hypothesis generation and evidence collection is already common in discovery agents. AutoDiscovery and ExperiGen use experimental feedback to guide hypothesis search, while Contrastive Discovery explicitly consider competing explanations. LLM-AutoSciLab also considers explicit observation along with competing explanations.The paper should explain what its observation-space representation adds beyond these approaches, as the current comparisons do not sufficiently establish this distinction. (Is it the explicit observation space schema that's the core contribution? If so, then the paper should clarify the generalizability of the chosen construction, and add the discovery agents with a similar discovery process but no explicit observation space schema to the baseline comparison)
  • The ablations and analysis doesn't sufficiently explain the performance gains. It's unclear what information remains when the observation space is removed. Removing comparative analysis also removes an investigation stage, so the effect may entangle differences in computation. And, improvements over CORAL seem to be mixed or minimal on several tasks, and the results are reported without uncertainty estimates. Therefore it's difficult to assess the true performance of the proposed method. Additionally, Figure 4 has missing legend in (b) and overlapping text. I see that the appendix version has the full legend, but it would be good to make sure the figure in the main text is still self-contained.
Questions for authors
  1. What's the main contribution of the observation space?
  2. How would SHELDON compare with closely related hypothesis discovery agents under a common task and budget?
  3. What remains available in the observation-space ablation? Are observations and evidence relations still accessible?
  4. What prompts, initial solutions, and fitness functions were used for EvoX? How were these choices validated?
  5. How consistent are the performance differences across repeated runs, particularly for the comparison between SHELDON and CORAL? Do the exploration and exploitation findings also hold across tasks and runs?
6 comments

We thank the reviewer for their useful feedback. SHELDON explicitly represents and independently explores the space of possible observations that helps in novel hypotheses, rather than treating evidence collection as an operation conditioned on existing hypotheses. We provide additional ablations and analyses that directly substantiate this distinction.

Novelty of Dual-Space Search and Distinction from Prior Discovery Agents

Reviewer: The novelty of dual-space search needs clearer grounding. Iterating between hypothesis generation and evidence collection is already common in discovery agents.

Existing agents, including AutoDiscovery, ExperiGen, Contrastive Discovery, and LLM-AutoSciLab, primarily score hypotheses, guiding subsequent searches based on those scores rather than actively seeking novel, unexplained observations. They are generally designed to work with supplied datasets or simulators, operate mainly on numerical data, and often require specific experimental interfaces.

SHELDON differentiates itself in three key ways:

  • Independent, Observation-Driven Discovery: SHELDON investigates data directly from observations without the need for an initial motivating hypothesis, enabling the generation of entirely novel hypotheses.
  • Explicit Representation of Observations: Observations are maintained as distinct records, and those that remain unexplained by any existing hypothesis are systematically tracked. SHELDON also maintains an evidence map, which records the relationships between all observations and hypotheses.
  • Candidate and Comparative Analysis: In contrast to existing agents that mainly focus on gathering supporting evidence, SHELDON systematically identifies both supporting and refuting evidence during candidate analysis. Furthermore, it enables comparative analysis by deriving additional insights through direct comparison of each hypothesis with the others.

Targeted Ablations

We conducted targeted ablations in which all settings remained unchanged except for the intervention specified in each ablation. The base model, task set, hypothesis-diversity instructions, candidate count, number of rounds, tool availability, and budget settings were held constant unless explicitly modified by the intervention. Actual expenditure was also comparable: approximately $2.10 per ORCA incident and 0.80–0.85 per DiagnosisArena case across all ablation variants. Thus, each comparison isolated the effect of a single mechanism while preserving the rest of the system configuration.

  • Disable new-observation search: Generate hypotheses using only the task description, without consulting the case material.
  • Remove new observation instruction: Omit the prompt that instructs the system to prioritize newer observations.
  • Attach observations directly to hypotheses: Keep observations in context alongside the hypotheses.

We observe that relative to the full-system of SHELDON, attaching observations directly to hypotheses gives the largest differences: −18.67 points on DiagnosisArena and −17.78 points on ORCA. Disabling independent observation search gives differences of −16.67 and −11.11 points, respectively. Removing the observation-seeking instruction gives differences of −10.67 and −11.11 points. This shows that all the components of SHELDON help the system.

AblationDiagnosisArena (%)ORCA RCA (%)
SHELDON (three-run mean)84.67**61.11 **
Disable new-observation search68.00 (−16.67)50.00 (−11.11)
Remove new observation instruction74.00 (−10.67)50.00 (−11.11)
Attach observations directly to hypotheses66.00 (−18.67)43.33 (−17.78)

The full SHELDON row presents three-run means: 84.67% on DiagnosisArena and 61.11% on ORCA, using the three-round setup. Parenthetical values indicate percentage-point differences from the full-system mean, calculated prior to rounding.

Why explicit instruction to look for new observations helps in exploration

We investigated why removing the explicit instruction to seek new observations negatively impacts SHELDON's performance. Our analysis shows that, despite access to the same set of tools, SHELDON makes significantly fewer tool calls to explore the case material when this instruction is omitted. This highlights simply providing tool availability and enabling iterative reasoning are not sufficient to ensure ongoing exploration of the case material/environment

On average per task, SHELDON used tools to discover new observations 10.8, 2.7, 1.4, and 1.2 times per round in Diagnosis Arena, and 8.2, 3.1, 2.3, and 2.0 times per round in ORCA-bench, whereas the variant without observation instruction did so 8.7, 0.2, 0, and 0 times in Diagnosis Arena, and 5.4, 0.3, 0.3, and 0 times in ORCA-bench.

Ablation Validity and Related Baselines

Reviewer: The ablations and analysis do not sufficiently explain the performance gains.

The comparative-analysis ablation disables the comparative mechanism while retaining explore, candidate analysis, and exploit stages. As in the other ablations, all settings except the named intervention remain fixed, with actual costs of approximately \2.10 per ORCA incident and \0.80- 0.85 per DiagnosisArena case. This intervention is separate from the four observation-space ablations summarized above.

Likewise, the observation-space ablation does not deprive the agent of evidence. All observations remain accessible within each round, while hypotheses, associated evidence, test notes, and reasons for discarding candidates persist across rounds. Only the explicit observation regions and global observation ledger are removed. The ablation therefore isolates the contribution of persistent, hypothesis-independent observational structure.

The suggested discovery agents differ in their assumed datasets, experimental interfaces, and investigation workflows. We therefore add a comparison with AutoDiscovery on ARC-AGI-2, where both methods can investigate rules from examples. Extending these agents to the other benchmarks would require additional adaptations whose effects would need to be distinguished from those of the published methods.

AutoDiscovery on ARC-AGI-2

We evaluated SHELDON and AutoDiscovery on the same 50 ARC-AGI-2 tasks, with both methods costing approximately $2.50 per task. This setup enables a direct, cost-controlled comparison of performance on a shared rule-discovery benchmark.

At the same task cost of $2.5, AutoDiscovery solves just 8.0% of the 50 ARC-AGI-2 tasks, compared to SHELDON's 46.0% (paired Wilcoxon, p < 0.0001). With an additional final answer call, AutoDiscovery improves to 23.0% (p = 0.002). We attribute SHELDON’s superior performance to its explicit strategy of seeking new observations and rigorously analyzing candidate hypotheses, rather than merely generating new hypotheses based on existing data and hypotheses as most of the discovery agents do.

Comparison with CORAL and Robustness

Reviewer: How consistent are the performance differences across repeated runs, particularly for SHELDON versus CORAL? Do the exploration and exploitation findings also hold across tasks and runs?

Response:

The repeated-run results below indicate that SHELDON achieves higher mean performance than CORAL on all four out of five ADRS tasks and tied on PRISM (Table 2 of submission). For optimization tasks, even small performance gains can be highly significant.

Notably, SHELDON also outperforms CORAL on four out of six molecular optimization tasks, identifying three best-in-library molecules. We want to emphasize the importance of the consistency of improvements that SHELDON shows, rather than solely their absolute score, especially for tasks nearing saturation.

Our analysis of repeated runs further highlights the consistency of SHELDON:

On Cloudcast, CORAL achieved a mean score of 619.00 (± 0.000), while SHELDON achieved 618.09 (± 0.0173); on LLMSQL, CORAL scored 0.7115 (± 0.004) compared to SHELDON’s 0.7204 (± 0.003); and on Transaction Scheduling, CORAL obtained a mean score of 4077 (± 107.39) versus SHELDON’s 4198 (± 65.82).

For each method, we performed three independent runs to evaluate consistency. On ADRS tasks, SHELDON consistently achieves higher mean performance than CORAL, with minimal run-to-run variability. These improvements are particularly stable on Cloudcast and LLMSQL, whereas Transaction Scheduling exhibits higher variability.

Exploration Across Tasks

We further evaluated explorative performance across six tasks using AIM's solution-novelty methodology, ensuring a matched evaluation cost (k = 20). SHELDON achieves 1.4× to 20.9× higher median hull share than the strongest baseline on five out of six tasks. The only exception is Cloudcast, where EvoX attains greater hull coverage, though SHELDON secures a slightly better best score (618.07 vs. 618.08) -- lower is better. We will add this analysis to our next version.

While the numerical differences may appear small due to the difficulty of these tasks and scores plateauing near the top of the range, SHELDON’s gains stem from the discovery of qualitatively novel strategies rather than incremental tuning of a single approach. For instance, on Transaction Scheduling, SHELDON introduced a best-insertion construction method combined with earlier local search techniques. On LLM_SQL, it identified a multiset-canonical layout and integrated this with recursive grouping, resulting in a program that runs in 4.3 seconds, compared to EvoX's 7.1 seconds (see Sections 5 and 6 of the submission).

EvoX Prompts, Initial Solutions, and Fitness Functions

Reviewer: What prompts, initial solutions, and fitness functions were used for EvoX? How were these choices validated?

Response:

We use the official EvoX and SkyDiscover libraries and their released task configurations. The prompts are unchanged, and both initial_program.py and evaluator/evaluator.py are byte-identical to the official GitHub versions. No custom initialization, fitness-function modification, or prompt tuning was introduced to favor SHELDON.

We will incorporate the targeted ablations, observation-activity analysis, repeated-run statistics, and novelty results into the revised manuscript, along with correcting Figure 4's legend and overlapping labels.

In summary, SHELDON's novelty is not another hypothesis-feedback loop. It makes observation discovery an independent search objective, explicitly tracks unexplored evidence, and preserves observations beyond individual hypotheses. The targeted ablations directly demonstrate the importance of these design choices.

Comments on the paper

1 comment

We thank all three reviewers for their constructive feedback. We are encouraged that the reviewers recognize several strengths of SHELDON: its very extensive evaluation, covering wide set of tasks and baselines, showing promising results across verifiable and non-verifiable tasks. They also appreciate our organization and adaptation of dual-space search to LLM agents, with the explicit organization of hypotheses and evidence making it easy to follow, along with good presentation of figures. We appreciate that they like the extensive reproducibility materials and trajectory analyses.

We have conducted substantial additional experiments directly showing the usefulness of having a separate observation space that should be explored independently of the hypotheses

Leakage: During the rebuttal preparation, we identified ground-truth leakage in the ORCA task-directory names: they contained a gold error-flag identifier visible to all methods. We reran SHELDON and all baselines after replacing those directory names with arbitrary identifiers. The corrected comparison is reported below.

After rerunning all the baselines and SHELDON. The performance of ReAct, and Codex, went up, from 40.00 to 46.67, and 40.00 to 50.00, respectively. For GOS, we found that the performance dropped to 23.33 from 30.00. Whereas, SHELDON and RLM remain the same. Further, we report additional metrics.

MethodRCA (%)RewardHallucination rateGold causes credited (%)Precision (cause level)Recall (cause level)Cost per task
ReAct46.670.6226.6756.882%56%0.45
Codex50.000.66810.0058.071%59%0.57
Codex Consistency43.330.6676.6761.776%64%1.72
Codex best of n (Oracle)56.670.7703.3374.179%73%1.72
Codex synthesis53.330.7913.3370.473%73%1.75
RLM40.000.64510.0055.663%56%0.39
GoS23.330.43336.6725.953%29%4.60
SHELDON66.670.8213.3380.287%80%2.21

RCA accuracy, hallucination rate, precision, and recall follow ORCA-Bench's original definitions; the reviewer-specific responses explain them. Reward is normalized RCA depth, using the benchmark verifier.

RCA: Root Cause Analysis Reward: Calculated using a rubric 0 - 3, where three means most complete answer, and 0 means completely hallucinated answer. The calculated reward for each task is then divided by 3. Hallucination Rate: The percentage of answers for each task that contains no correct error flag, i.e, all the error flags are hallucinated. Precision: Of all the root causes a report names, the percentage that are gold causes. Pooled across 30 tasks. Recall: Of all the root cause events across all 30 tasks, the percentage of causes credited to each method. Cost per task: The mean spend on model calls per task, in USD.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.