- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes SHELDON, an LLM agent harness for investigative reasoning, built on the hypothetico-deductive method and the dual-space search model of Klahr & Dunbar (1988). The harness keeps explicit hypothesis and observation spaces, each divided into axes and regions, and links observations to hypotheses with categorical evidence relations (support, refute, neutral, unresolved). In each round the agent explores, analyzes each hypothesis, compares hypotheses with one another, and then keeps, repairs, refines, or shelves them, with a cooldown on shelved ones. Using GPT-5.6-terra, the authors evaluate on medical diagnosis (DiagnosisArena), incident root-cause analysis (a modified ORCA-Bench), ARC-AGI-2, QNN circuit topology, DOCKSTRING, and system-optimization tasks (ADRS). They report gains over the baselines, notably on DiagnosisArena (84.00 vs 78.00) and ORCA-Bench (66.67 vs 40.00). Ablations show drops of up to 14 points, and SHELDON covers more ground-truth causes than GoS.
Critical strengths
- Grounding the harness in Klahr & Dunbar's dual-space search is coherent, and generalizing the experiment space to an observation space (Sec. 4.1) is a reasonable adaptation for agentic settings.
- The task set covers problems with and without a scoring function (diagnosis, root-cause analysis, ARC, program optimization) in one harness.
- Reproducibility material is extensive (prompts, state structures, hyperparameters, task IDs, per-method cost), and Sec. 6 goes beyond headline scores with cause-coverage analysis and trajectory case studies.
Critical weaknesses
- It is unclear whether the gains come from the dual-space structure or from extra compute. SHELDON spends
$2.91per DiagnosisArena case and$6.90per ORCA-Bench incident, while ReAct, Codex and RLM spend$0.02-$1.56(Table 5). The comparison with GoS at similar cost helps, but no single-agent baseline gets the same budget through generic test-time scaling, such as best-of-N, self-consistency, or parallel rollouts with an aggregator like RSA. Without that, the paper's main claim, that explicit hypothesis/observation structure is what helps, is not isolated. - The ORCA-Bench setup rewards breadth. The authors changed the task to ask for "all plausible causes" and dropped causes that started after the reporting time (App. D). The verifier counts an incident as solved when the report names every ground-truth cause, so a harness that keeps and reports many hypotheses has a natural advantage. The changes apply to all methods, but precision (non-ground-truth causes per report) is not given. The same concern applies to the 76.6% vs 46.8% coverage analysis.
- The contribution of each design element is not isolated. GoS, Arbor and ScientistOne already maintain hypothesis-evidence structures. SHELDON adds the axes/regions map, the comparative stage and the shelf/cooldown, but no ablation compares the axes/regions map with plain observation logging, and Table 3 does not say what "w/o observation space" removes. SHELDON also delegates implementation to GPT-5.6-luna, while the baselines use terra only. Two descriptions do not match the implementation: the PROVISIONAL attribute (Sec. 4.2) is missing from the Observation dataclass (App. E), and the QNN "human baseline" is never reported.
- The statistical evidence is thin. The DiagnosisArena (n = 50) and ORCA-Bench (n = 30) gaps are 3 and 8 cases. The tables appear to be single runs with no standard deviations. On ADRS, several tasks are ties or near-ties at saturation. The paper does not state the number of ARC-AGI-2 tasks or the scoring rule.
Questions for authors
- Can you compare against compute-matched generic scaling (best-of-N, self-consistency or RSA with ReAct/Codex) at about
$2.9and$6.9per task? - For which kinds of tasks should an explicit hypothesis/observation space help, and when might it hurt (e.g., by anchoring on early hypotheses)? Do the results across scored and unscored tasks support such a characterization?
- On ORCA-Bench, what is the precision of each method's report, and what are the results on the unmodified task?
- What does the axes/regions map add compared with a flat log of observations, and what exactly does "w/o observation space" remove? How much of the gain remains with a terra-only configuration?
- Is PROVISIONAL implemented, and what is the human baseline for QNN topology?
7 comments
2. ORCA Evaluation and Root-Cause Precision (W2, Q3)
Reviewer: The ORCA-Bench setup rewards breadth. On ORCA-Bench, what is the precision of each method's report, and what are the results on the unmodified task?
To clarify the setup: the original ORCA-Bench task already states that there may be multiple root causes, and its verifier already requires every gold cause to be named. We did not add causes to the gold set. We made two changes, applied identically to all methods. First, we added a line to the task description clarifying this requirement; the requirement itself is already part of the original task. Second, we dropped gold causes that began after the incident was reported, since an operator could not have observed them at that time; this only shrinks the gold set. We will make this wording precise in Appendix D, and we report results on the unmodified benchmark below.
ORCA-Bench Metric Definitions
We use the original definitions in ORCA-Bench, Appendix H. RCA accuracy requires every gold cause; hallucination rate flags a nonempty report with no correct cause. Precision is the fraction of extracted causes that are correct (zero for no attributions); recall is the fraction of gold causes recovered. Values are averaged over incidents. A flag without a variant matches any gold variant. Thus, high RCA accuracy and low hallucination rate can coexist with extra incorrect causes; precision addresses that concern.
Reward is normalized RCA depth: each incident's mean 0–3 rubric score divided by three, then averaged across incidents. Incorrect or misleading claims reduce rubric scores. This follows the original verifier's scoring and reward computation.
Original ORCA Setup
We also ran the unmodified benchmark, with the original instruction and all gold causes, for SHELDON and the strongest baseline, Codex. Both methods score lower than in our setup (Codex 26.67% vs. 50.00%; SHELDON 33.33% vs. 66.67%), consistent with the issues above affecting all methods alike. SHELDON still leads on RCA accuracy (33.33% vs. 26.67%) and on cause-level precision (93% vs. 77%).
| Method | RCA (%) | Reward | Hallucination rate | Gold causes credited (%) | Precision (cause level) | Recall (cause level) | Cost per task |
|---|---|---|---|---|---|---|---|
| Codex | 26.67 | 0.564 | 10.00 | 41.1 | 77% | 43% | 0.61 |
| SHELDON | 33.33 | 0.570 | 3.33 | 51.6 | 93% | 49% | 1.90 |
Four-Round ORCA Metrics
Under our modified task setup, the following four-round table reports 87% cause-level precision for SHELDON and its best-of-three RCA accuracy of 66.67%. These precision and hallucination-rate values are not measurements of the lower-budget round-3 checkpoint. Reporting precision together with recall and reward addresses whether higher coverage is accompanied by additional incorrect attributions. The separate robustness section reports the three-run comparison with the strongest original single-agent baseline.
| Method | RCA (%) | Reward | Hallucination rate | Gold causes credited (%) | Precision (cause level) | Recall (cause level) | Cost per task |
|---|---|---|---|---|---|---|---|
| ReAct | 46.67 | 0.622 | 6.67 | 56.8 | 82% | 56% | 0.45 |
| Codex | 50.00 | 0.668 | 10.00 | 58.0 | 71% | 59% | 0.57 |
| Codex Consistency | 43.33 | 0.667 | 6.67 | 61.7 | 76% | 64% | 1.72 |
| Codex best of n (Oracle) | 56.67 | 0.770 | 3.33 | 74.1 | 79% | 73% | 1.72 |
| Codex synthesis | 53.33 | 0.791 | 3.33 | 70.4 | 73% | 73% | 1.75 |
| RLM | 40.00 | 0.645 | 10.00 | 55.6 | 63% | 56% | 0.39 |
| GoS | 23.33 | 0.433 | 36.67 | 25.9 | 53% | 29% | 4.60 |
| SHELDON | 66.67 | 0.821 | 3.33 | 80.2 | 87% | 80% | 2.21 |
We will add these results to our next draft.
3. Isolating the Observation-Space Contribution (W3, Q4)
Reviewer: The contribution of each design element is not isolated.
Response:
Axes and regions are one supporting component; the main contribution is the process of discovering observations, generating hypotheses from them, and rigorously testing those hypotheses. SHELDON searches for observations without requiring an existing hypothesis to motivate the investigation. Unexpected or unexplained observations can therefore suggest new mechanisms and improve the hypotheses under consideration. Candidate analysis investigates each explanation, and comparative analysis seeks evidence that distinguishes competing explanations. The accumulated evidence determines which hypotheses are retained, revised, or discarded.
We conducted targeted ablations in which all settings remained unchanged except for the intervention specified in each ablation. The base model, task set, hypothesis-diversity instructions, candidate count, number of rounds, tool availability, and budget settings were held constant unless explicitly modified by the intervention. Actual expenditure was also comparable: **approximately \0.80–0.85 per DiagnosisArena case across all ablation variants. Thus, each comparison isolated the effect of a single mechanism while preserving the rest of the system configuration.
- Disable new-observation search: Generate hypotheses using only the task description, without consulting the case material.
- Remove new observation instruction: Omit the prompt that instructs the system to prioritize newer observations.
- Attach observations directly to hypotheses: Keep observations in context alongside the hypotheses.
We observe that relative to the full-system of SHELDON, attaching observations directly to hypotheses gives the largest differences: −18.67 points on DiagnosisArena and −17.78 points on ORCA. Disabling independent observation search gives differences of −16.67 and −11.11 points, respectively. Removing the observation-seeking instruction gives differences of −10.67 and −11.11 points. This shows that all the components of SHELDON help the system.
| Ablation | DiagnosisArena (%) | ORCA RCA (%) |
|---|---|---|
| SHELDON | 84.67 | 61.11 |
| Disable new-observation search | 68.00 (−16.67) | 50.00 (−11.11) |
| Remove new observation instruction | 74.00 (−10.67) | 50.00 (−11.11) |
| Attach observations directly to hypotheses | 66.00 (−18.67) | 43.33 (−17.78) |
The full SHELDON row presents three-run means: 84.67% on DiagnosisArena and 61.11% on ORCA, using the three-round setup. Parentheses values indicate percentage-point differences from the full-system mean, calculated prior to rounding.
The observed pattern points to larger differences for independent observation search and evidence retention than for the axes/regions representation alone; these differences are descriptive, without uncertainty estimates for the ablation variants.
Our "Attach observations directly to hypotheses" ablations directly address the reviewer's question about structured-memory alternatives.
These results distinguish SHELDON from simply maintaining structured notes or refining hypotheses. The independent observation search allows the agent to investigate directions not motivated by existing hypotheses.
Importantly, the original (in the submission) observation-space ablation does not remove evidence access. Observations remain available within each round; hypotheses, associated evidence, test notes, and reasons for discarding candidates persist across rounds. Only the global observation ledger and explicit observation regions are removed.
Similarly, removing comparative analysis does not eliminate investigation: explore, candidate analysis, and exploit remain intact. The ablations therefore target specific mechanisms rather than removing the entire discovery process.
Model Allocation
For ORCA, Diagnosis Arena, ARC-AGI-2 and QNN, we use terra for all our experiments. Only ADRS uses Luna for delegated implementation within SHELDON, which is a weaker model than GPT-5.6-terra (used by the baselines). Thus, SHELDON's gains cannot be attributed simply to using a stronger implementation model. In fact, SHELDON achieves higher performance despite delegating implementation to a weaker model.
We agree that a terra-only SHELDON configuration would provide an additional useful control. We plan to investigate the effect of implementation-model choice in the next iteration and will clarify the model allocation and its limitations in the revised manuscript.
We thank the reviewer for their insightful feedback. SHELDON's central contribution is actively finding new observations that lead to new and better-supported hypotheses, while rigorously testing candidate hypotheses against the evidence.
1. Generic Scaling and Computational Cost (W1, Q1)
We expanded Codex’s computation through independent investigations and answer aggregation. Caching issues were resolved, and SHELDON's hyperparameters were revised for tasks lacking scoring functions. Four computation rounds were used. The previously reported four-round result now incurs an average cost \0.83perDiagnosisArenacaseand2.21 per ORCA incident. We also report the round-3 ORCA checkpoint, averaging \$1.55 per incident. All comparisons use the revised configuration.
Rollout and aggregation protocol.
ORCA: Run three independent Codex rollouts. For aggregation, a separate Codex instance reviews the investigation trajectories and final reports from all three rollouts to produce the final answer. DiagnosisArena: Run five independent Codex investigations in parallel, prompting each toward a \0.15per−rolloutspendingtarget.Aggregateoutputsfromallfiveruns.Thefinalcostmaydifferfromtherealizedexpenditure:themeasuredcostforthescaledvariantsis0.65 per case, as reported in the table.
We evaluated three answer-combination strategies:
Lower-Budget ORCA Comparison
Round 3 SHELDON at an average cost of \$1.55 per incident. Across three runs, it gets a 57.8% mean RCA, with mean reward 0.729. These are averages across evaluation runs. The scaled Codex baselines below, each use three independent rollouts per incident and were evaluated once.
RCA (Root-Cause Accuracy) means if the agent found all the root cause analysis for a task.
At this operating point, SHELDON has 4.4 percentage points higher mean RCA accuracy at approximately 11% lower cost than Codex aggregationReward at round 3 is lower than both aggregation and oracle selection, so the result shows a accuracy–cost advantage. The scaled baselines have not been repeated.
Four-Round and DiagnosisArena Results
SHELDON's accuracy values in the following table, 88.00% on DiagnosisArena and 66.67% on ORCA, are the best of three runs. We report all three SHELDON runs and their means separately in the robustness section. The scaled Codex strategies were each evaluated once. The cost columns describe the reported per-run expenditure; selecting SHELDON's best-of-three score requires three evaluations.
RCA means Root-Cause Accuracy – if the agent found all the root cause analysis for a task. Reward: Calculated using a rubric 0 - 3, where three means most complete answer, and 0 means completely hallucinated answer. The calculated reward for each task is then divided by 3.
The lower-budget comparison above is our direct response to the ORCA compute concern. The table here retains the four-round and DiagnosisArena results with their original reporting conventions. SHELDON's three-run means at four rounds are 61.1% on ORCA and 84.7% on DiagnosisArena; the 66.67% and 88.00% values in this table are best-of-three scores. The paired comparisons in the robustness section provide the available repeated-run evidence against the original single-agent baselines.
We will add these results to the next version of the paper.