acceptodds
Under review as a conference paper at ICLR 2027

OracleResolve: Hypothesis-Guided Evidence Reasoning for Oracle Bone Script Decipherment

Abstract

Oracle bone script (OBS) decipherment requires reasoning across heterogeneous evidence, including glyph morphology, inscriptional context, diachronic character forms, and philological scholarship. Existing computational approaches increasingly incorporate such evidence, yet fixed evidence pipelines and generic tool-routing agents do not explicitly couple competing interpretations with the selection of subsequent evidence. This limitation is particularly important in OBS decipherment, where multiple morphologically plausible readings may require different contextual, historical, or philological evidence to distinguish, and where the correct interpretation may be absent from the initial candidate set. We introduce OracleResolve, a hypothesis-guided research agent that maintains candidate interpretations as persistent hypotheses together with supporting evidence, contradicting evidence, scores, and unresolved questions. Morphology-based Glyph and Component Experts first construct the initial hypothesis set from the target glyph. During subsequent reasoning, OracleResolve identifies unresolved disagreements among the current hypotheses and selectively queries contextual, diachronic, or philological evidence. Retrieved evidence is explicitly evaluated to revise the hypothesis state and subsequent investigation; when the accumulated evidence cannot adequately support the current candidate set, an open-candidate retrieval module expands the hypothesis space with new interpretations. We evaluate OracleResolve on a class-ID-disjoint OBS decipherment setting with an additional stricter reading-unseen subset, while preventing evaluation source groups from entering the contextual retrieval corpus. Controlled comparisons against direct multimodal reasoning, a fixed evidence pipeline, and a generic tool-routing agent show that OracleResolve improves decipherment reliability, better recovers from incorrect initial preferences and incomplete hypothesis spaces, without increasing average evidence-tool usage. These results support hypothesis-guided evidence reasoning as a promising framework for computational ancient-script decipherment.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Oracle Bone Script Decipherment, LLM Agent, Multimodal Reasoning, Hypothesis-Guided Evidence Reasoning.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZWdtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OracleResolve maintains candidate oracle-bone readings with supporting evidence, counterevidence, scores, and unresolved questions. A central controller selects contextual, diachronic, or philological evidence to distinguish candidates, revises their ranking, and expands the candidate set when necessary. The paper describes 88 cases from 44 held-out character classes and a 68-case reading-unseen subset, with evaluation source groups excluded from contextual retrieval. Matched-tool comparisons report 26.83% Top-1 accuracy and 2.67 evidence calls per case for the proposed method, alongside recovery and component ablations. These reported results require clarification of evaluation denominators.

Critical strengths
  1. The persistent hypothesis state and distinction between supporting and contradicting evidence are well suited to an ambiguous historical interpretation task. Candidate expansion addresses a genuine limitation of fixed-set reranking, and the paper compares against both a fixed pipeline and a generic agent using shared tools and initial proposals.
  2. The evaluation design explicitly addresses two important sources of leakage: character-ID overlap and reuse of the same inscriptional source group. It additionally reports a reading-unseen subset because class-ID separation alone does not guarantee unseen reading strings. These controls are consequential for assessing the task. Critical Weaknesses
Critical weaknesses
  1. The reported accuracies are incompatible with the declared instance-level denominators under the stated metric. With N=88, Top-1 accuracy must change in increments of about 1.136 percentage points, so 26.83% is not a possible rounded single-run value; 22.58% is likewise incompatible with N=68. Fixed-Pipeline's 11.90% also does not fit N=88. No alternative averaging or method-specific failure-exclusion rule is documented in Appendix H. The reported 4.10-point gain is not interpretable as a matched, complete-panel single-run comparison until the numerators, denominators, aggregation protocol, and failure handling are clarified. This inconsistency does not by itself identify the underlying cause.
  2. Validation is too limited to establish reliable evidence reasoning for decipherment. The test uses printed/transcribed glyphs with accepted readings, only 44 classes, and a third-party model identifier without a pinned backend snapshot or repeated-run uncertainty. Correct reading strings do not evaluate whether support/counterevidence judgments, source quotations, or disputed interpretations are valid; the paper has no independent expert evaluation of those outputs. This weakens claims of decipherment reliability and evidence efficiency, for which tool-call count also omits controller token and inference costs.
Questions for authors
  1. Please provide success counts and evaluation denominators for every Table 1 row, including all parse failures, abstentions, and unresolved cases. Are results averaged over runs, computed on different evaluated subsets, or aggregated by another rule? Please document the protocol and, if subsets differ, also report comparisons on the same complete 88- and 68-case panels.
  2. Were all methods given byte-identical initial hypotheses for each instance, and why do the recovery percentages appear to use different eligible subsets? Please provide the fixed subset counts and paired per-case outcomes.
  3. What independent assessment, if any, verifies the evidence judge's stance assignments and source-grounded reasoning? Please distinguish known-reading identification from genuinely unresolved character decipherment.
3 comments

Thank you for identifying the evaluation inconsistency and for recognizing the value of persistent hypotheses and source separation. We have rerun the complete evaluation and distinguish below the corrected results, the scope of the task, and the remaining limitations.

Q1 / W1: Exact counts, denominators, and failure handling.

The original aggregation excluded records that failed after repeated timeouts, resulting in method-specific denominators. For example, 26.83% was 22/82 and 11.90% was 10/84; the corresponding reading-unseen counts were 14/62 and 4/64. This was an aggregation error, not a rounding issue. The revised results come from a complete rerun, rather than merely changing the denominators of the old predictions. Every method is now evaluated on the fixed 88-case panel and the same 68-case reading-unseen subset.

On the full panel, the exact Top-1 / Top-3 counts are: Direct-LLM, 1/88 and 4/88; Fixed-Pipeline, 12/88 and 26/88; Generic-Agent, 20/88 and 26/88; OBS-Expert (adapted), 23/88 and 33/88; and OracleResolve, 25/88 and 33/88. On the reading-unseen subset, they are respectively 1/68 and 1/68; 6/68 and 14/68; 13/68 and 18/68; 14/68 and 20/68; and 17/68 and 21/68.

Cases without valid predictions remain in the denominator and receive zero accuracy credit; a missing valid ranking contributes zero to MRR. An unresolved case with a valid final ranking is scored using that ranking, while its unresolved status is retained separately. MRR averages reciprocal ranks and therefore does not have an integer success-count numerator. We made these rules explicit in Appendix H and synchronize the numerical statements with the corrected tables.

Q2: Shared initialization and fixed recovery subsets.

Fixed-Pipeline, Generic-Agent, and OracleResolve load the same cached initial hypotheses and evidence for each instance, including candidate identities, ordering, and initial scores. Recovery strata are defined once from this cache, before evaluating subsequent reasoning outcomes. The original aggregation excluded different failed records for different methods, which changed the evaluated strata; the revised aggregation retains every eligible instance.

The full panel contains 7 initially correct Top-1 cases, 9 cases with an accepted reading in the initial set but not ranked first, and 72 cases without any accepted reading in the initial set. For OracleResolve, rank recovery is 3/9, erroneous revision is 0/7, expansion recall is 23/72, and expansion Top-1 is 15/72. Its final 25 correct predictions decompose as 7 - 0 + 3 + 15. For Generic-Agent, the corresponding final count is 7 - 1 + 1 + 13 = 20.

Q3 / W2: Independent evidence validation and the meaning of decipherment.

We have not completed an independent assessment of the judge's stance assignments or of the semantic and linguistic validity of the generated explanations. A support or contradiction judgment contributes to scoring only when its quotation can be matched verbatim to the retrieved material. This checks quotation existence and traceability.

Our quantitative task is recovery of documented reference readings from transcribed or printed glyph crops. The private reference labels are withheld at runtime, local expert training class IDs are disjoint from test IDs, and evaluation source groups are excluded from contextual retrieval. The 68-case subset additionally excludes accepted reading strings that overlap with local expert training labels.

W2: Model reproducibility, repeated runs, and computational cost.

The requested and returned service identifier is gpt-6-sol, with reasoning effort high. As disclosed in Appendix E.1, the third-party service does not expose a dated backend snapshot, so the underlying model version and routing cannot be independently pinned. The complete rerun corrects the evaluation panel; it does not constitute an assessment of variability across multiple independent full runs. We do not currently report such between-run uncertainty.

The additional token accounting includes repeated requests. These requests consume resources and should not be discarded merely because they are retries. The average accounting is shown below: Direct-LLM 8,228.41; Fixed-Pipeline 54,445.72; Generic-Agent 64,879.68; OBS-Expert (adapted) 77,411.80; OracleResolve 73,347.77.

Reviewer_ZWdtOpenReview ↗

Response

Thank you for the detailed rebuttal and the additional experiments. The authors have addressed several important concerns, particularly the evaluation denominator inconsistencies, baseline comparisons, and the operational details of hypothesis revision and evidence weighting. The addition of OBS-Expert (adapted) also provides a useful reference point. However, the empirical advantage over this baseline remains marginal, with only two additional Top-1 correct cases and no improvement in Top-3 accuracy. More importantly, the central claim of evidence-guided interpretation remains insufficiently validated, as the authors acknowledge that neither the semantic correctness of the generated explanations nor the evidential support of the cited passages has been independently assessed by domain experts. The current evaluation primarily measures recovery of documented readings rather than genuine decipherment of previously unresolved characters. Overall, the rebuttal improves the experimental rigor and clarifies the scope of the contribution, but the remaining limitations prevent me from changing my original assessment. I therefore maintain my rating.

Thank you for your follow-up. Since our previous response, we have completed an expert-rating pilot to address the previously missing assessment of final-report quality. Two evaluators specializing in classical philology, one PhD student and one master's student, independently assessed the method-anonymized final reports from OracleResolve and OBS-Expert (adapted). Neither evaluator was involved in system development, and they completed their ratings without communicating with each other. Evaluating on five-point scales for interpretive correctness (IC), evidential support (ES), and research usefulness (RU).

To reduce fatigue and ensure consistent evaluation conditions, we select nine cases for human evaluation. The nine selected cases comprise three where both methods are Top-1 correct, two where only OracleResolve is correct, two where only the adapted baseline is correct, and two where both are incorrect. Both methods therefore identify five of the nine readings correctly. Averaged across cases and raters, OracleResolve scores 3.44/3.44/3.11 on IC/ES/RU, compared with 2.67/2.83/2.22 for the adapted baseline. We provide the rating protocol and complete per-case scores in the new appendix section, “Expert Assessment of Final Reports.”

This adds expert judgments of final explanations and their presented evidential support to the previous reading-level evaluation. We hope this additional evidence helps clarify both the contribution and its remaining limits.

Reviewer_DBVCOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

OracleResolve is a hypothesis-guided agent for oracle-bone-script (OBS) decipherment that maintains competing readings, retrieves evidence to resolve disagreements and expands the candidate set when needed. Experiments on 88 instances report improved accuracy and hypothesis recovery over fixed-pipeline and generic-agent baselines.

Critical strengths
  1. The framework addresses both incorrect initial rankings and missing candidates through hypothesis revision and expansion, a well-motivated design for evidence-intensive decipherment.
  2. The evaluation uses matched baselines that share the initial hypotheses and evidence resources, helping isolate the contribution of the reasoning strategy. It also separates training and evaluation class IDs, excludes evaluation source groups from contextual retrieval, and reports a stricter reading-unseen subset.
Critical weaknesses
  1. The reported accuracies are inconsistent with the stated evaluation protocol. Under the instance-level metric in Appendix H, 26.83% cannot correspond to an integer number of correct predictions over 88 instances; similarly, 22.58% is incompatible with 68 instances. This inconsistency affects the central performance claims and prevents verification of the reported improvement.
  2. The specification of key reasoning decisions remains incomplete in the manuscript. Although prompt templates and some decision rules are provided, the criteria for prioritizing disagreements and assigning evidence-reliability weights are not sufficiently clear. These decisions directly affect evidence acquisition and candidate ranking.
  3. The evaluation provides limited support for the decipherment claim. It includes 88 instances from 44 classes and a stricter reading-unseen subset of 68 instances. While this subset strengthens the generalization test, it does not expand the overall evaluation, and the reported gains may depend on a few cases. Moreover, evaluation uses transcribed or printed glyphs with established readings, without independent expert assessment of the generated evidence reports.
  4. The comparison with related methods is limited. Although the paper discusses differences from AlphaOracle and OracleAgent, it evaluates only internally constructed fixed-pipeline and generic-agent baselines. It remains unclear how faithfully these baselines represent prior workflows and whether OracleResolve improves upon them.
Questions for authors
  1. How do the accuracies in Table 1 reconcile with the stated sample counts and the metric definition in Appendix H? Please provide exact success counts and denominators, or explain how these values were calculated.
  2. What operational rules govern disagreement selection, evidence-reliability assignment, and hypothesis revision? Which decisions are explicitly implemented, and which are left to the LLM controller?
  3. What validation supports the correctness of the generated evidence reports beyond reference-reading accuracy? Have domain experts independently assessed whether the cited evidence supports the proposed interpretations?
  4. What evidence establishes that this small test set is sufficient to demonstrate the method's effectiveness and that the reported gains are not driven by a few individual cases?
  5. How do Fixed-Pipeline and Generic-Agent represent the workflows of AlphaOracle or other related methods? Please clarify their correspondence and explain why no direct or adapted comparison with these closely related systems is provided.
4 comments

Thank you for the detailed questions about evaluation validity, operational decisions, and baseline correspondence. We have rerun the complete evaluation, added a task adaptation of OBS-Expert, and clarify the supported conclusions below.

Q1 / W1:

The original aggregation incorrectly excluded records that failed after repeated timeouts. Thus, 26.83% was 22/82 and 11.90% was 10/84, while the corresponding reading-unseen counts were 14/62 and 4/64. This was an aggregation error. The revised results are from a complete rerun on the fixed 88-case panel and its fixed 68-case subset, not merely a denominator correction of the old outputs.

On the full panel, Top-1 / Top-3 counts are: Direct-LLM, 1/88 and 4/88; Fixed-Pipeline, 12/88 and 26/88; Generic-Agent, 20/88 and 26/88; OBS-Expert (adapted), 23/88 and 33/88; and OracleResolve, 25/88 and 33/88. On the reading-unseen subset, the corresponding counts are 1/68 and 1/68; 6/68 and 14/68; 13/68 and 18/68; 14/68 and 20/68; and 17/68 and 21/68.

Cases without valid predictions remain in the denominator and receive no accuracy credit. Missing rankings contribute zero to MRR. An unresolved status does not automatically invalidate an otherwise valid final ranking; such cases are scored by the same ranking rules and retain their status separately.

Q2 / W2:

The LLM proposes a disagreement, a tool, and a query; deterministic code validates these proposals and maintains the numerical state. The program identifies the leading candidates from their current scores. The planner is prompted to compare observable differences between them, register an expected observation for each, and select a query that could test those expectations. A newly registered question must cover the current top two candidates, and their prediction descriptions must not be identical. When an existing question is revisited, its registered predictions cannot be changed retrospectively. These checks enforce coverage and record consistency.

Source-reliability coefficients are assigned by resource records and tool adapters, rather than by the judge. The reported coefficients are 0.5 for newly acquired glyph references and inscriptional examples, 0.25 for unverified automatic component predictions, and zero additional weight for initialization materials. The lower component coefficient discounts an additional automatic inference step; initialization is not counted again because it already affects the initial prior. These are fixed design choices, not calibrated probabilities. The judge supplies stance, strength, and relevance, while code checks source quotations, combines these fields with source reliability, groups evidence by provenance, and updates scores. We state the numerical update and the implemented stopping and expansion rules explicitly, keeping these operations distinct from LLM-generated planning decisions.

Q3 / W3: Generated evidence reports.

We have not completed an independent evaluation of explanation semantics, linguistic validity, or whether each cited passage actually supports the associated interpretation. Our automatic check requires a verbatim source quotation before support or contradiction can affect a candidate's score. It verifies quotation existence and traceability, not entailment. The present quantitative findings therefore concern reference-reading prediction and candidate recovery.

Q4 / W3: Test-set size and the improvement.

The full panel contains 44 classes with two instances per class. The reading-unseen subset contains 34 of these classes and 68 instances; it is a stricter subset, not additional independent test data. Equal class counts avoid domination by more frequent classes. Context availability also varies: 3 cases have no context, 35 have 1-3 context characters, and 50 have at least 4.

Against Generic-Agent, OracleResolve is correct on 25/88 versus 20/88 cases. In the paired outcomes, only OracleResolve is correct on six cases, and only Generic-Agent is correct on one. The six gains occur in six different classes; the reverse outcome occurs on the other instance of one of those classes. Hence net gains occur in five classes, rather than being concentrated in a single class.

Q5 / W4:

Fixed-Pipeline and Generic-Agent are controlled strategy baselines inspired by AlphaOracle's staged evidence integration and OracleAgent's task-driven tool planning, respectively. Their models, tools, and resources are implemented in our evaluation environment. They are not end-to-end reproductions of the original systems, and the results do not establish superiority over those complete systems. We will specify the retained, replaced, and omitted mechanisms in Appendix G rather than implying equivalence through a method name.

Direct end-to-end reproduction was constrained by implementation availability: the OracleAgent repository we checked remained marked "coming soon", and the AlphaOracle materials we could access were insufficient to verify a complete local reproduction.

Thank you for the detailed response and additional experiments. Rerunning the complete evaluation and reporting exact success counts and denominators largely resolves Q1/W1, which was our primary concern. The distinction between programmatic rules and LLM decisions together with the explanation of evidence-reliability weights also clarifies the method. The addition of OBS-Expert also provides a useful reference point. Your comment have resolve most of my concerns. However, the gains over OBS-Expert remain somewhat small, and the small evaluation set still limits conclusions about generalizability. Besides,the lack of independent validation of the generated explanations and their evidential support also leaves the broader decipherment claim insufficiently validated.

Overall, I appreciate that your responses address most of my concerns, so I will raise my score.

Thank you for reassessing the revised experiments and method specification. To address your remaining concern about explanation-level validation, we have now added an expert-rating pilot involving two domain experts specializing in classical philology, one PhD student and one master's student, and nine paired cases, with method names concealed. Neither evaluator was involved in system development, and they completed their ratings without communicating with each other.

The experts rated the final reports of OracleResolve and OBS-Expert (adapted) on five-point scales for interpretive correctness, evidential support, and research usefulness. The respective mean scores are 3.44 versus 2.67, 3.44 versus 2.83, and 3.11 versus 2.22. Both methods are Top-1 correct on five of these nine selected cases, so the difference in average ratings is not attributable to OracleResolve having a greater number of correct predictions in this subset.

The appendix reports the selection composition, scoring protocol, and all individual ratings. This pilot supplies new expert judgments beyond reference-reading accuracy.

Thank you for your reply, which addresses some of my remaining concerns. I will maintain my updated rating.

Reviewer_uNEGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes OracleResolve, an agent that threat oracle bone script interpretation as iterative hypothesis testing. Experiments on 88 instances from 44 held-out character classes report improved reading accuracy and candidate recovery over direct LLM inference, fixed-pipeline, and generic-agent baselines, with ablations examining individual components.

Critical strengths
  1. The writing is clear and easy to follow.
  2. Explicit tracking of supporting and contradicting evidence provides a useful structure for inspecting the reasoning process.
Critical weaknesses
  1. Insufficient comparison with foundational work. The paper should compare against Specializing Large Models…, AlphaOracle, and OracleAgent. These methods provide relevant foundations and baselines for testing whether explicit hypothesis competition improves interpretation. Comparisons with custom baselines alone do not establish this added value.
  2. Metrics do not assess interpretation quality. Character-identity and recovery metrics overlook semantic consistency, linguistic validity, and evidence quality. They cannot establish whether the system produces more credible interpretations.
  3. Insufficient validation for practical decipherment. Evaluation covers previously interpreted characters with substantial available evidence. No dedicated tests on genuinely undeciphered or evidence-scarce characters, or independent expert validation, are reported.
Questions for authors
  1. How do the custom baselines relate to Zhang et al. (2026), AlphaOracle, and OracleAgent? Are they intended as faithful adaptations, or were there specific barriers to comparing with these methods?
  2. Is any evaluation of the generated explanations available beyond character accuracy, particularly regarding semantic consistency, linguistic validity, and evidential support?
  3. Could you clarify the intended scope of “decipherment”? Do you have results on evidence-scarce or genuinely undeciphered characters, or feedback from domain experts, that help establish the framework’s practical value?
3 comments

Thank you for highlighting the relationship to prior OBS systems and the distinction between character-identity accuracy and interpretation quality. We have added the requested Zhang et al.'s component-grounded comparison and clarify the task and evaluation scope below.

Q1 / W1: Relationship to prior systems and the new OBS-Expert adaptation.

Fixed-Pipeline and Generic-Agent are controlled implementations inspired by AlphaOracle's staged integration of evidence and OracleAgent's task-driven tool planning, respectively. They use our evaluation environment to compare reasoning strategies. We do not present them as complete reproductions of the original systems or infer that their results establish superiority over the original systems. Appendix G will make the retained mechanisms, replaced models and tools, and omitted task components explicit.

Direct end-to-end reproduction was constrained by implementation availability: the OracleAgent repository we checked remained marked "coming soon", and the AlphaOracle materials we could access were insufficient to verify a complete local reproduction.

Following your suggestion, we have added a task adaptation of Zhang et al.'s component-grounded method, named OBS-Expert (adapted). It retains component-explanation retrieval, component-to-character relations, and the separation of evidence organization from semantic reasoning. Our test cases do not include verified component segmentations, so we replace the original segmented-component input and DINOv2 prototype classifier with the local automatic multilabel component predictor. This predictor does not itself produce component boundaries or spatial relations. The adaptation receives the target image and available target-position context, excludes the reference character identity and reference-identity fallback lookup, and predicts ranked character identities with brief component-based explanations. Retrieval is executed programmatically, followed by staged LLM processing.

On the complete 88-case panel, OBS-Expert (adapted) achieves 23/88 Top-1 and 33/88 Top-3, compared with OracleResolve's 25/88 and 33/88. On the fixed 68-case reading-unseen subset, it achieves 14/68 and 20/68, compared with 17/68 and 21/68. Failures and abstentions are not removed from the evaluation denominator. The full-set Top-1 difference is two cases, and Top-3 is tied. We interpret this as a comparison under the stated task adaptation, rather than a comparison with the original paper's interpretation-generation performance. The adaptation's resource visibility, component-label mapping, initialization, and budgets will be specified in Appendix G.

Q2 / W2: Semantic, linguistic, and evidential evaluation.

We agree that reference-reading accuracy and candidate-recovery metrics do not establish explanation quality. We have not completed an independent evaluation of the semantic consistency or linguistic validity of the generated explanations, or of the correctness of their evidence-to-point relationships. Our implemented source check requires a support or contradiction judgment to include a quotation that can be matched verbatim to the retrieved material before it contributes to candidate scoring. This verifies quotation existence and traceability, but does not prove that the quoted passage supports the interpretation.

Accordingly, our empirical conclusion is that hypothesis-guided evidence acquisition and revision improve reference-reading prediction in the stated controlled evaluation. We will distinguish inspectable evidence records from independently validated interpretations and remove broader statement of established interpretive reliability. Scholarly opinions available to the system as retrieval material are not treated as an independent expert assessment of its generated outputs.

Q3 / W3: Scope of decipherment and evidence of practical value.

The experiment evaluates recovery of documented readings from transcribed or printed glyph crops, with private reference labels withheld at runtime. Local expert training class IDs and test class IDs are disjoint, and evaluation source groups are globally excluded from contextual retrieval. Reading-Unseen is the 68-case subset for which no accepted reference-reading string overlaps with a local expert training label.

This controlled setup permits objective scoring of reference-reading recovery while studying candidate competition and expansion. The framework records competing hypotheses, supporting and contradicting materials, and unresolved questions for inspection; its practical paleographic value still requires experts' validation. We will state this scope consistently in the abstract, evaluation protocol, and conclusion.

I appreciate the authors’ response. However, I still believe the result in my Q2 is critical: it cannot just be an empirical conclusion. You may check the paper by Zhang et al. (2026); they provided a different level of analysis. Even within an annotated dataset, he can still find a plausible way to comprehensively analyze the system’s performance on oracle decipherment tasks. Also, for Q3, undeciphered characters are not an excluded group... According to my collaboration with oracle experts, they only care about what insight our model can bring to them to decipher those undeciphered characters. They never care about those already deciphered. So at least talk with them about the usefulness of your work. I think this is the basic standard for an ICLR paper. So I will maintain my score.

Thank you for clarifying the distinction between reference-reading accuracy and the validity and usefulness of generated explanations. Following your comments, we have completed an additional expert-rating pilot on nine selected cases. Two evaluators specializing in classical philology, one PhD student and one master's student, independently assessed the method-anonymized final reports from OracleResolve and OBS-Expert (adapted). Neither evaluator was involved in system development, and they completed their ratings without communicating with each other. They use separate five-point scales for interpretive correctness (IC), evidential support (ES), and research usefulness (RU).

The mean IC/ES/RU scores are 3.44/3.44/3.11 for OracleResolve and 2.67/2.83/2.22 for the adapted baseline. Both methods correctly identify five of the nine readings. In the three cases where both methods are correct, their mean IC and ES ratings are equal (4.00 and 3.67, respectively), while the mean RU ratings are 3.67 for OracleResolve and 3.00 for the adapted baseline. This provides a descriptive assessment of report quality beyond counting character matches.

Regarding Q2, the new appendix section, “Expert Assessment of Final Reports,” provides the scoring protocol, selection composition, and complete individual ratings. The assessment covers the final explanations and their presented evidence; it does not evaluate intermediate candidates or revision steps that are absent from the final reports.

Regarding Q3, the usefulness ratings concern assistance with examining these selected, documented-reading cases. We do not present them as evidence of successful decipherment or demonstrated research progress on genuinely undeciphered characters. We appreciate your feedback in motivating this additional assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions and evidence for hypothesis-guided reasoning

Dear Area Chair and Reviewers,

Thank you for the discussion. OracleResolve addresses a question that arises when interpreting an ambiguous glyph: which evidence would distinguish the current hypotheses, and when does that evidence warrant proposing a new one? The agent keeps competing hypotheses active, uses their disagreements to guide retrieval, and can reopen the candidate set as evidence accumulates.

The revised evaluation shows why this distinction matters. In 72 of 88 cases, the initial proposals omit every accepted reading, so even a perfect reranker restricted to those proposals could solve at most 16 cases. OracleResolve ranks an accepted reading first in 25/88 cases, compared with 23/88 for OBS-Expert (adapted) and 20/88 for Generic-Agent. On the 72 cases requiring a reading outside the initial set, the corresponding counts are 15, 14, and 13. OracleResolve also retains all seven initially correct hypotheses.

We added an independent assessment of the final reports. Two evaluators specializing in classical philology rated the reports with method names concealed. We selected nine cases covering all four combinations of correct and incorrect predictions, with five correct predictions per method by design. OracleResolve received higher mean ratings than OBS-Expert (adapted) for interpretive correctness (3.44 vs. 2.67), evidential support (3.44 vs. 2.83), and research usefulness (3.11 vs. 2.22) on five-point scales.

The paired ratings show what character accuracy alone leaves open. Where both methods identified an accepted reading, both evaluators rated OracleResolve’s report more useful in two of three cases and equally useful in the third. Where only OracleResolve was correct, both evaluators rated its reports higher on all three dimensions in both cases. Where only the baseline was correct, both evaluators still gave the reports equal usefulness scores in both cases. Section 4.4 and Appendix J provide the protocol and individual ratings.

These findings suggest one reason to maintain competing hypotheses: investigating what distinguishes them may yield evidence a researcher can use even when the agent’s final choice is wrong. We believe OracleResolve contributes to the study of tool-using agents by showing how disagreements among hypotheses can guide evidence collection and by evaluating both the readings the agent recovers and the usefulness of its reports to researchers. Thank you for considering the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.