- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
OracleResolve maintains candidate oracle-bone readings with supporting evidence, counterevidence, scores, and unresolved questions. A central controller selects contextual, diachronic, or philological evidence to distinguish candidates, revises their ranking, and expands the candidate set when necessary. The paper describes 88 cases from 44 held-out character classes and a 68-case reading-unseen subset, with evaluation source groups excluded from contextual retrieval. Matched-tool comparisons report 26.83% Top-1 accuracy and 2.67 evidence calls per case for the proposed method, alongside recovery and component ablations. These reported results require clarification of evaluation denominators.
Critical strengths
- The persistent hypothesis state and distinction between supporting and contradicting evidence are well suited to an ambiguous historical interpretation task. Candidate expansion addresses a genuine limitation of fixed-set reranking, and the paper compares against both a fixed pipeline and a generic agent using shared tools and initial proposals.
- The evaluation design explicitly addresses two important sources of leakage: character-ID overlap and reuse of the same inscriptional source group. It additionally reports a reading-unseen subset because class-ID separation alone does not guarantee unseen reading strings. These controls are consequential for assessing the task. Critical Weaknesses
Critical weaknesses
- The reported accuracies are incompatible with the declared instance-level denominators under the stated metric. With N=88, Top-1 accuracy must change in increments of about 1.136 percentage points, so 26.83% is not a possible rounded single-run value; 22.58% is likewise incompatible with N=68. Fixed-Pipeline's 11.90% also does not fit N=88. No alternative averaging or method-specific failure-exclusion rule is documented in Appendix H. The reported 4.10-point gain is not interpretable as a matched, complete-panel single-run comparison until the numerators, denominators, aggregation protocol, and failure handling are clarified. This inconsistency does not by itself identify the underlying cause.
- Validation is too limited to establish reliable evidence reasoning for decipherment. The test uses printed/transcribed glyphs with accepted readings, only 44 classes, and a third-party model identifier without a pinned backend snapshot or repeated-run uncertainty. Correct reading strings do not evaluate whether support/counterevidence judgments, source quotations, or disputed interpretations are valid; the paper has no independent expert evaluation of those outputs. This weakens claims of decipherment reliability and evidence efficiency, for which tool-call count also omits controller token and inference costs.
Questions for authors
- Please provide success counts and evaluation denominators for every Table 1 row, including all parse failures, abstentions, and unresolved cases. Are results averaged over runs, computed on different evaluated subsets, or aggregated by another rule? Please document the protocol and, if subsets differ, also report comparisons on the same complete 88- and 68-case panels.
- Were all methods given byte-identical initial hypotheses for each instance, and why do the recovery percentages appear to use different eligible subsets? Please provide the fixed subset counts and paired per-case outcomes.
- What independent assessment, if any, verifies the evidence judge's stance assignments and source-grounded reasoning? Please distinguish known-reading identification from genuinely unresolved character decipherment.
3 comments
Response
Thank you for the detailed rebuttal and the additional experiments. The authors have addressed several important concerns, particularly the evaluation denominator inconsistencies, baseline comparisons, and the operational details of hypothesis revision and evidence weighting. The addition of OBS-Expert (adapted) also provides a useful reference point. However, the empirical advantage over this baseline remains marginal, with only two additional Top-1 correct cases and no improvement in Top-3 accuracy. More importantly, the central claim of evidence-guided interpretation remains insufficiently validated, as the authors acknowledge that neither the semantic correctness of the generated explanations nor the evidential support of the cited passages has been independently assessed by domain experts. The current evaluation primarily measures recovery of documented readings rather than genuine decipherment of previously unresolved characters. Overall, the rebuttal improves the experimental rigor and clarifies the scope of the contribution, but the remaining limitations prevent me from changing my original assessment. I therefore maintain my rating.
Thank you for your follow-up. Since our previous response, we have completed an expert-rating pilot to address the previously missing assessment of final-report quality. Two evaluators specializing in classical philology, one PhD student and one master's student, independently assessed the method-anonymized final reports from OracleResolve and OBS-Expert (adapted). Neither evaluator was involved in system development, and they completed their ratings without communicating with each other. Evaluating on five-point scales for interpretive correctness (IC), evidential support (ES), and research usefulness (RU).
To reduce fatigue and ensure consistent evaluation conditions, we select nine cases for human evaluation. The nine selected cases comprise three where both methods are Top-1 correct, two where only OracleResolve is correct, two where only the adapted baseline is correct, and two where both are incorrect. Both methods therefore identify five of the nine readings correctly. Averaged across cases and raters, OracleResolve scores 3.44/3.44/3.11 on IC/ES/RU, compared with 2.67/2.83/2.22 for the adapted baseline. We provide the rating protocol and complete per-case scores in the new appendix section, “Expert Assessment of Final Reports.”
This adds expert judgments of final explanations and their presented evidential support to the previous reading-level evaluation. We hope this additional evidence helps clarify both the contribution and its remaining limits.
Thank you for identifying the evaluation inconsistency and for recognizing the value of persistent hypotheses and source separation. We have rerun the complete evaluation and distinguish below the corrected results, the scope of the task, and the remaining limitations.
Q1 / W1: Exact counts, denominators, and failure handling.
The original aggregation excluded records that failed after repeated timeouts, resulting in method-specific denominators. For example, 26.83% was 22/82 and 11.90% was 10/84; the corresponding reading-unseen counts were 14/62 and 4/64. This was an aggregation error, not a rounding issue. The revised results come from a complete rerun, rather than merely changing the denominators of the old predictions. Every method is now evaluated on the fixed 88-case panel and the same 68-case reading-unseen subset.
On the full panel, the exact Top-1 / Top-3 counts are: Direct-LLM, 1/88 and 4/88; Fixed-Pipeline, 12/88 and 26/88; Generic-Agent, 20/88 and 26/88; OBS-Expert (adapted), 23/88 and 33/88; and OracleResolve, 25/88 and 33/88. On the reading-unseen subset, they are respectively 1/68 and 1/68; 6/68 and 14/68; 13/68 and 18/68; 14/68 and 20/68; and 17/68 and 21/68.
Cases without valid predictions remain in the denominator and receive zero accuracy credit; a missing valid ranking contributes zero to MRR. An unresolved case with a valid final ranking is scored using that ranking, while its unresolved status is retained separately. MRR averages reciprocal ranks and therefore does not have an integer success-count numerator. We made these rules explicit in Appendix H and synchronize the numerical statements with the corrected tables.
Q2: Shared initialization and fixed recovery subsets.
Fixed-Pipeline, Generic-Agent, and OracleResolve load the same cached initial hypotheses and evidence for each instance, including candidate identities, ordering, and initial scores. Recovery strata are defined once from this cache, before evaluating subsequent reasoning outcomes. The original aggregation excluded different failed records for different methods, which changed the evaluated strata; the revised aggregation retains every eligible instance.
The full panel contains 7 initially correct Top-1 cases, 9 cases with an accepted reading in the initial set but not ranked first, and 72 cases without any accepted reading in the initial set. For OracleResolve, rank recovery is 3/9, erroneous revision is 0/7, expansion recall is 23/72, and expansion Top-1 is 15/72. Its final 25 correct predictions decompose as 7 - 0 + 3 + 15. For Generic-Agent, the corresponding final count is 7 - 1 + 1 + 13 = 20.
Q3 / W2: Independent evidence validation and the meaning of decipherment.
We have not completed an independent assessment of the judge's stance assignments or of the semantic and linguistic validity of the generated explanations. A support or contradiction judgment contributes to scoring only when its quotation can be matched verbatim to the retrieved material. This checks quotation existence and traceability.
Our quantitative task is recovery of documented reference readings from transcribed or printed glyph crops. The private reference labels are withheld at runtime, local expert training class IDs are disjoint from test IDs, and evaluation source groups are excluded from contextual retrieval. The 68-case subset additionally excludes accepted reading strings that overlap with local expert training labels.
W2: Model reproducibility, repeated runs, and computational cost.
The requested and returned service identifier is
gpt-6-sol, with reasoning efforthigh. As disclosed in Appendix E.1, the third-party service does not expose a dated backend snapshot, so the underlying model version and routing cannot be independently pinned. The complete rerun corrects the evaluation panel; it does not constitute an assessment of variability across multiple independent full runs. We do not currently report such between-run uncertainty.The additional token accounting includes repeated requests. These requests consume resources and should not be discarded merely because they are retries. The average accounting is shown below: Direct-LLM 8,228.41; Fixed-Pipeline 54,445.72; Generic-Agent 64,879.68; OBS-Expert (adapted) 77,411.80; OracleResolve 73,347.77.