- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PaST-Q is a patient-modeling and multi-agent framework for guideline-grounded cancer treatment planning. Patient records are organized into temporally ordered Observation–Action blocks with unordered events within blocks; role-specific set encoders and a causal Transformer produce a shared patient state S_t. A Future-Query predictor answers requests of the form (role, horizon) directly — a query token combines a role embedding, a time embedding, and the current state, feeds shared cross-attention and role-specific experts, and predicts the future latent of the requested role, with a persistence-plus-residual prediction head. Candidate treatments are appended as Action blocks and the frozen encoder re-computes the state, yielding treatment-conditioned forecasts at 7/30/90 days. Lightweight readouts translate latents into clinical statements; a learned ranker combines a historical treatment-choice prior with predicted-outcome features; a multi-agent layer adds guideline retrieval and synthesizes recommendations. Evaluation covers history-integration and future-simulation ablations on MMRF CoMMpass (public) and a 6,449-patient private cervical-cancer cohort, latent-forecasting and readout benchmarks, a full-system agent benchmark, and two small physician studies. Ethics and AI-use statements are complete.
Critical strengths
S1 — Exemplary evidence discipline for a clinical ML system. Patient-level disjoint splits; a frozen preprocessing pipeline that respects when information actually becomes available (retrospective summaries timestamped at completion); explicit accounting of supervision eligibility per horizon; persistence controls at every evaluation level; component-wise ablations that isolate history integration and future simulation separately; and scrupulously honest reporting of every cell where the method is second best on the public cohort. This protocol makes the paper's measured claims trustworthy, and it is a template worth propagating.
S2 — A reusable architectural abstraction with practical throughput. The future-queryable patient state — encode the history once, then answer arbitrary (role, horizon) queries, including candidate-treatment-conditioned ones, without retraining the encoder for new readout tasks — is a clean interface that other clinical decision-support systems can adopt. The resulting agent completes all 200 benchmark tasks at a mean of 1.76k LLM tokens, which, whatever the comparison caveats, is a practically meaningful cost profile.
Critical weaknesses
W1 — The counterfactual framing exceeds the evidence, and the core validity question is unaddressed. Predicting outcomes under treatments a patient never received, from observational data, is a causal-inference problem; the paper cites the treatment-effect literature (Bica et al.; Melnychuk et al.; Lim) but engages none of its machinery — no identifiability assumptions, no overlap diagnostics, no off-policy evaluation, no negative controls. The deployed metric (agreement with recorded treatment) cannot distinguish a correct dynamics model from a physician-mimicry prior, and Eq. (5)'s log p_choice term is never ablated from the residual. For a system pitched as patient-specific treatment comparison, this silence is consequential.
W2 — Reproducibility and statistical support are below the paper's evidentiary standard. The training objective is prose-only with all four λ weights missing; there is no hyperparameter table, no code release, no MMRF cohort statistics, and no candidate-pool sizes for the retrieval metrics; no confidence intervals anywhere; and the physician studies (18 and 8 assessments) are framed as validation when they are feasibility anecdotes — the largest reported gain rests on three assessments. These are fixable, but they currently block independent verification of the main quantitative claims.
Questions for authors
Q1 (clarification, affects the central claim). What evidence distinguishes the candidate-treatment simulations from confounded observational prediction? Can you provide any proxy validation — e.g., calibration or ranking accuracy restricted to patient subgroups who actually received each candidate, negative-control outcomes, or overlap/positivity diagnostics for the candidate pool?
Q2 (clarification, affects Eq. (5)). How much of the ranking performance is carried by the historical treatment-choice prior log p_choice versus the outcome-feature residual R_ψ? An ablation separating the two terms would clarify what future simulation contributes to the final recommendation.
Q3 (clarification, affects reproducibility). Can you provide the formulas for the five loss terms in Eq. (3), the values of λ_c, λ_Δ, λ_r, λ_o, the target-window size, the training hyperparameters, and a code release for the MMRF pipeline (which uses only public data)?
Q4 (clarification, affects Table 6). How many distinct target candidates enter the retrieval ranking for each role and cohort, and can you report confidence intervals for Tables 1, 2, and 6?
Q5 (clarification, affects Section 4.4). Given 18 and 8 paired assessments, will you reframe the physician studies as feasibility evidence, or expand them? As written, a single changed assessment flips the Q1 comparison.
Q6 (suggestion, future work). A comparison with causal treatment-effect models (e.g., the Causal Transformer line you cite) on overlapping patient subgroups would connect this system to the literature whose problem it adopts.
Response to Reviewer fqXM
Thank you for valuable feedback! We will address your concerns below and hope they resolve your issues.
Response to W2 and Q3 can be found in general response G1, response to W1 can be found in G2, and response to Q1 can be found in G3.
Q2: Clarification on Eq. (5)
We added a score-term ablation of Eq. (5) on 179 patients from the supplementary Private-CC benchmark. With the model and candidate sets fixed, we compared prior-only, residual-only, and combined scores. Top-1 evaluates treatment-component agreement. The results are shown below.
The prior provides the main ranking foundation, and adding the residual raises Top-1 by 1.68 percentage points and F1 by 2.92 points. The residual combines state, candidate, and forecast features to refine this ranking. Table 2 separately evaluates future simulation: enabling it raises F1 from 58.10% to 60.65% on Private-CC and from 72.02% to 72.73% on MMRF. Together, these results show how forecast-informed refinement complements the historical choice prior.
Q4: Confidence intervals.
For Table 6, the full Observation/Action retrieval pools contain 606/108 candidates for MMRF and 18,099/3,123 for Private-CC, shared across methods and horizons. We have added 95% confidence intervals for the PaST-Q results below.
Tables 1 and 2: treatment recommendation (%, estimate [95% CI]).
Table 6: future-latent forecasting (estimate [95% CI]).
Q5: About physician assessment.
We present both physician studies as feasibility and exploratory evidence. Their value is in proving our method receives more favorable assessments, and could provide additional assistance to the physicians. We will review the paper to ensure that the claims consistently reflect this exploratory scope and avoid overclaiming.
Q5: About suggestions on future work.
Thank you for this constructive suggestion. Comparing with causal treatment-effect models such as Causal Transformer is a valuable next step.