- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes CARTER, a method for partially repairing taxonomy trees under noisy feedback. It retains a calibrated set of plausible trees and certifies only changes supported by every remaining tree, providing opposing example trees for unresolved decisions.
Critical strengths
The paper identifies an important and interesting failure mode: incorrect feedback can eliminate the correct taxonomy while increasing agreement among the remaining candidates in sec 2. This motivates preserving plausible alternatives rather than treating agreement as sufficient evidence for repair. CARTER addresses this problem by allowing a calibrated number of feedback disagreements and certifying only changes shared by all retained trees in sec 2 . In the simulated-feedback study in sec 3, it adds approximately 2.1 certified edits per session over no feedback, while producing an incorrect edit in 2 of 148 sessions. These results support its ability to make useful repairs while limiting errors.
Critical weaknesses
The practical evaluation is limited. Feedback comes from simulated answerers in sec 3, and CARTER makes no repairs on the six genuine taxonomy transitions in sec 6. It therefore remains unclear whether the method can make useful repairs with real feedback, rather than stay safe by avoiding changes.
The guarantee requires the correct taxonomy to be among the candidates. However, when candidate recall drops to 94.2%, the reported session-level wrong-certificate rate reaches 41.9% in sec 6. This raises concerns about reliability when candidate generation misses correct parent relations.
Questions for authors
The purpose and relevance of several implementation details in the appendix are difficult to understand. The authors should explain how these details support reproducibility or the paper’s scientific claims. Examples include:
- Lines 1010–1013 report an artifact-set SHA-256 hash and distinguish development checks from sealed-holdout evidence, but their relevance to the surrounding discussion is unclear.
- Lines 1024–1025 provide a session identifier digest without clearly explaining what it identifies or how readers should use it.
- Similar clarification is needed for lines 1105–1107, 1593–1597, and 1620–1622.
Response to XEjm: Practical evidence, candidate failure, and appendix navigation
Thank you for identifying target retention as the key safety issue and recognizing the useful edits in the programmed-feedback study. We address the practical evidence, candidate failure, and appendix navigation below.
The 148-session programmed-feedback study adds 2.088 canonical-correct certified edits per session over no feedback, while a lifetime wrong edit occurs in 2/148 sessions. This measures useful committed repair under the declared simulated answer regime. The separate 224-taxonomy structural holdout evaluates repair without feedback.
On the six genuine MeSH/OEWN transitions, CARTER emits zero edits. The retained allowed graphs contain 68.6–100% of complete candidate edges, leaving no unanimous actionable change. This is the interface's deferral behavior under broad uncertainty, and its zero-error rate measures abstention. Genuine changes paired with real noisy feedback remain a distinct empirical question; the positive structural and programmed-feedback results establish utility in their respective tested settings.
The reported 0.941588 recall is child-weighted edge recall. The theorem's premise is whole-tree inclusion: every target parent edge must be available. In this stress test, 142 of 2,431 true-parent edges are removed, and initial whole-tree containment falls to 0.041. Average edge recall therefore does not measure the event protected by the guarantee.
The 0.419 wrong-session rate is a substantive failure: trees that all omit a true parent can still agree on an incorrect replacement or deletion. A calibrated feedback budget cannot restore an edge excluded upstream. The main experiments use complete candidates within each local node set, ensuring target inclusion by construction and isolating the calibration layer. Application to a restricted generator requires a separate audit of whole-tree inclusion and adequate candidate expansion; the stated guarantee applies when the target-containing candidate premise holds. The stress test makes this prerequisite measurable rather than treating per-edge recall as sufficient evidence of reliability.
The hashes identify specific experimental objects so a reader can match a reported result to its saved inputs or outputs. The scientific support comes from result tables, exactness checks, and paired implementation parity. Each identifier has a different role:
Lines 1010–1013: the artifact-set identifier in results/liar_query_dev_v1c/metadata.json binds the recorded trial, oracle, summary, and README files. Design and graph metadata have separate fields. The surrounding distinction tells readers that these are query/transcript implementation checks, with a different evidential role from the sealed structural holdout.
Lines 1024–1025: the digest identifies the sorted 148 reliability-calibration session IDs for iid_panel/ambiguity_block, recorded in results/joint_e2e_public_dev_v1/reliability_budgets.csv. It identifies which sessions selected the feedback budget of one; it is distinct from the 148 evaluation sessions and from a digest of their transcripts.
Lines 1105–1107: the identifier refers to the historical GNN multiseed aggregate result manifest. The released numerical table is results/public_gnn_multiseed_dev_v1/graph_seed_averaged.csv, which supports reaggregation of the learned-scorer results.
Lines 1593–1597: the two identifiers refer to the frozen configuration payload and output artifact manifest in results/adjoint_compiler_scaling_dev_v2b/. The configuration identifies the scaling grid; the manifest identifies saved measurements, summaries, and parity records. The relevant evidence for shared computation is agreement with the reference alongside the matched timing results.
Lines 1620–1622: this identifies the historical results/compiler_scaling_dev_v1c/ reference-compiler/certificate bundle underlying Table 11. It assigns those timings and declared computation caps to that implementation; the current shared compiler is evaluated separately.
For reproduction, start with REPRODUCE.md, select the study and its CSV/README, then use the associated identifier to match the saved object. A digest alone supplies identity, rather than evidence of accuracy or a fresh experimental replay.
Together, the useful-edit results and candidate stress test establish the capability and its operational prerequisite: calibrated partial repair over a target-containing family, with unresolved alternatives exposed through witnesses. We would appreciate reconsideration of the soundness assessment in light of this distinction.