- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
CRAFT frames traditional Chinese medicine prescription recommendation as stepwise set construction. It encodes symptom–herb relations via hierarchical graph reasoning, combines a neural compatibility score with a co-occurrence-based marginal symptom-coverage utility (CMU), and learns a STOP action to decide prescription length. A holistic scoring branch and final calibration layer sit on top of the trajectory scores. Evaluation uses TCM PD and TCM SMGCN, with fixed-cutoff ranking metrics as the primary measures.
Critical strengths
- The stepwise construction with CMU-guided expansion is a natural fit for prescription generation, where both the set of herbs and the size of the set matter clinically. The writing is clear, and the graph reasoning plus CMU formulation is well explained.
- The ablation study and Figure 3 together show that CMU matters to the trained model: the inference-time perturbation produces clear drops, and the coverage function's properties are well illustrated.
Critical weaknesses
- The released code selects checkpoints by test F1@10 and uses test scores for early stopping. If this procedure produced the main tables, the reported results are not proper held-out evaluations. On top of that, the public scoring path is missing the holistic branch and final calibration described in the paper, so which model was actually evaluated is uncertain. Finally, the fixed-K metrics in the main tables don't measure the quality of prescriptions actually terminated by the learned STOP; the Table 3 examples are illustrative but not a dataset-level evaluation of this central feature.
- The experiments don't convincingly show why dynamic construction is needed over simpler alternatives. Thresholded multilabel models and sequential decoders already produce variable-size sets, but the paper lacks a matched comparison isolating the benefit of the learned construction process. CMU and dynamic reasoning contribute unevenly across datasets. Figure 3 shows the coverage function's properties and the trained model's reliance on CMU, but this doesn't explain why the retrained ablation effects on TCM PD are small. The clinical interpretation of CMU should also stay consistent with its stated definition — a co-occurrence-based, non-causal score.
AI-use disclosure. I could not find the required disclosure section in the manuscript or appendix.
Questions for authors
- Which data splits were used for checkpoint selection and early stopping? If the released procedure generated the results, can you re-evaluate with a proper validation split? Do different K values end up selecting different epochs?
- Which exact code version produced Tables 1–4? Where are the holistic branch and final calibration in that version? The README entry point and optimizer also don't match the manuscript — can you reconcile them?
- What are the actual set quality and length errors of prescriptions terminated by STOP, across each dataset?
- How does the learned construction compare with a matched thresholded multilabel model or a representative sequential decoder on the same TCM data?
- How much of CRAFT's performance can a CMU-only greedy selector recover?
- On TCM PD, the retrained CMU ablation shows small effects while the inference-time perturbation shows large ones, and dynamic reasoning contributes differently on SMGCN. How do you explain these discrepancies?
4 comments
Additional Response (1/2): Implementation Consistency and Evaluation Reliability
We thank the reviewer for carefully examining our released implementation. We would like to clarify the code-version discrepancy and report the results of our completed re-evaluation.
1. Clarification regarding the released code (W1/Q2)
We acknowledge that the previously released repository snapshot was not synchronized with the full architecture and evaluation procedure described in the manuscript.
In particular, the released version does not contain the complete holistic prescription branch, dynamic–global fusion, or final inference calibration. Its checkpoint-selection procedure also relies on test-set metrics.
We apologize for this inconsistency. We agree that the released implementation, in its current form, is insufficient to reproduce the complete method described in the paper.
We are preparing an updated repository release containing the complete model implementation, training and inference entry points, configuration files, and evaluation scripts.
Importantly, we distinguish this code-release issue from the results of our newly conducted experiments. The revised results are obtained using an explicit validation-based evaluation protocol, as described below.
We will also provide a code-to-result mapping identifying the implementation, checkpoints, and logs associated with the revised experimental results. Until this provenance is fully documented, we do not claim that the existing public snapshot reproduces all results in Tables 1–4.
2. Revised validation protocol (W1/Q1)
We conducted new experiments on both datasets using three random seeds (2021, 2022, and 2023).
The original training partitions were divided into fitting and validation subsets, while preserving the original test sets.
| Dataset | Fit | Validation | Test |
|---|---|---|---|
| TCM_PD | 18,638 | 2,071 | 13,056 |
| TCM_SMGCN | 20,625 | 2,292 | 3,443 |
Model optimization and training-derived association statistics use the fitting data.
Checkpoint selection and early stopping are based exclusively on validation Micro-F1@10.
A single checkpoint is selected for each run and subsequently used to evaluate all recommendation cutoffs (K = 5, 10, and 20).
Therefore, different values of K do not select different epochs in the revised experiments.
The test set is not used for checkpoint selection or early stopping.
3. Updated experimental results
The revised three-seed CRAFT results are:
| Dataset | Macro-F1@10 |
|---|---|
| TCM_PD | 0.2454 ± 0.0004 |
| TCM_SMGCN | 0.2516 ± 0.0005 |
These results replace the previous single-run estimates when evaluating CRAFT under the corrected validation protocol.
We acknowledge that the previous test-based checkpoint selection was methodologically inappropriate, and we do not use those original test-selected results as evidence of held-out generalization.
4. Reproducibility and reporting
We will distinguish revised experimental results from historical baseline results, and document the corresponding training configurations, checkpoint-selection procedure, and evaluation scripts.
We also acknowledge that correcting the public repository and establishing complete provenance for
Additional Response (1/2): Implementation Consistency and Evaluation Reliability
We thank the reviewer for carefully examining our released implementation. We would like to clarify the code-version discrepancy and report the results of our completed re-evaluation.
1. Clarification regarding the released code (W1/Q2)
We acknowledge that the previously released repository snapshot was not synchronized with the full architecture and evaluation procedure described in the manuscript.
In particular, the released version does not contain the complete holistic prescription branch, dynamic–global fusion, or final inference calibration. Its checkpoint-selection procedure also relies on test-set metrics.
We apologize for this inconsistency. We agree that the released implementation, in its current form, is insufficient to reproduce the complete method described in the paper.
We are preparing an updated repository release containing the complete model implementation, training and inference entry points, configuration files, and evaluation scripts.
Importantly, we distinguish this code-release issue from the results of our newly conducted experiments. The revised results are obtained using an explicit validation-based evaluation protocol, as described below.
We will also provide a code-to-result mapping identifying the implementation, checkpoints, and logs associated with the revised experimental results. Until this provenance is fully documented, we do not claim that the existing public snapshot reproduces all results in Tables 1–4.
2. Revised validation protocol (W1/Q1)
We conducted new experiments on both datasets using three random seeds (2021, 2022, and 2023).
The original training partitions were divided into fitting and validation subsets, while preserving the original test sets.
| Dataset | Fit | Validation | Test |
|---|---|---|---|
| TCM_PD | 18,638 | 2,071 | 13,056 |
| TCM_SMGCN | 20,625 | 2,292 | 3,443 |
Model optimization and training-derived association statistics use the fitting data.
Checkpoint selection and early stopping are based exclusively on validation Micro-F1@10.
A single checkpoint is selected for each run and subsequently used to evaluate all recommendation cutoffs (K = 5, 10, and 20).
Therefore, different values of K do not select different epochs in the revised experiments.
The test set is not used for checkpoint selection or early stopping.
3. Updated experimental results
The revised three-seed CRAFT results are:
| Dataset | Macro-F1@10 |
|---|---|
| TCM_PD | 0.2454 ± 0.0004 |
| TCM_SMGCN | 0.2516 ± 0.0005 |
These results replace the previous single-run estimates when evaluating CRAFT under the corrected validation protocol.
We acknowledge that the previous test-based checkpoint selection was methodologically inappropriate, and we do not use those original test-selected results as evidence of held-out generalization.
4. Reproducibility and reporting
We will distinguish revised experimental results from historical baseline results, and document the corresponding training configurations, checkpoint-selection procedure, and evaluation scripts.
We also acknowledge that correcting the public repository and establishing complete provenance for
Thank you for the response and the additional experiments. I still have reservations about the method’s overall contribution to prescription recommendation and the strength of the evidence for its benefits. In my view, the paper falls short of the ICLR bar, so I’ll keep my original rating.
I also noticed that Part 1 of your November 18 response appears to have been posted twice, and I haven’t seen Part 2 in the thread.
We thank Reviewer YFSN for the close reading of both the paper and the released code. The points raised are exactly the ones that most need strengthening, and we address them below.
This point deserves a direct fix rather than a clarification. In the revision we will report results from models re-trained with an explicit held-out validation split, with checkpoint selection and early stopping driven by validation F1@10 only. A single checkpoint will be selected per run and all values of K reported from that checkpoint, so K can no longer influence which epoch is selected. The updated training logs and evaluation scripts will be released together with the revision, and any change in the numbers will be reported transparently.
We apologize for the inconsistency between the released code and the paper. We will align them fully: the holistic branch (Section 3.5) and the final calibration layer (Section 3.7) will be included in the public repository, the README entry point and optimizer setting will be corrected to match Appendix A.3, and the exact commit that produced Tables 1–4 will be tagged and cited in the revision.
These will be reported in the revision as part of the adaptive-termination evaluation described above (length MAE and P/R/F1 of STOP-terminated prescriptions per dataset), so that the central feature of the method is measured at the dataset level rather than only through the two illustrative cases.
We will add this matched comparison in the revision: a thresholded multilabel model (tuned per patient or per dataset) and a representative sequential decoder, evaluated under the same splits and protocol, to isolate the benefit of the learned order-free construction process itself.
We will report the CMU-only greedy selector as a dedicated baseline on both datasets, quantifying how much of CRAFT's performance comes from the utility signal alone versus the neural compatibility and dynamic state tracking.
We view these as consistent rather than contradictory. The retrained ablation lets the model redistribute reliance onto the holistic/global branch during training, which partially compensates for the missing CMU input; the inference-time intervention instead isolates a dependence that was actually learned under CMU, so its effect is larger. The cross-dataset difference reflects the varying strength of the residual-demand signal relative to the graph-based representations on each dataset. We will add this discussion to Section 4.5 in the revision.
We have added the disclosure to Appendix A.4: "No AI tools were used in the writing of this paper," consistent with our OpenReview declaration.