- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes PsyCAT for self-report computerized adaptive testing. It combines explicit response-style modelling, Gaussian posterior mixtures, shared-attention post-calibration, minimax aggregation, and information-regularized item selection. Evaluation includes held-out response prediction on six questionnaire banks, controlled latent-change experiments, and an expert study of information leakage.
Critical strengths
- The paper addresses response-style variation and information leakage in self-report CAT through an integrated estimation and selection framework.
- PsyCAT achieves the lowest mean endpoint RPS among the evaluated methods across six questionnaire banks, providing consistent evidence of improved response prediction.
Critical weaknesses
- The stated objective is to minimize trait-estimation error, but the main real-data evaluation focuses on held-out response prediction. No direct comparison between CAT trait estimates and full-questionnaire scores or estimates is reported. This limits the evidence for improved assessment accuracy.
- The evidence for trait–style separation remains model-dependent. The controlled latent-change experiments use generators derived from fitted response models, while the style model captures threshold-spacing preferences. The expert study evaluates sequence inferability rather than measurement reactivity.
Questions for authors
See the weakness section.
3 comments
Thank you for your additional experimental results and comments.
- W1: I think that the comparison with the full-questionnaire results are useful and it solves most of my concern. However, I think the common GRM/EAP is used to compare the trait estimated value and it doesn't directly evaluate the effectiveness of original trait estimation method.
- W2: I think that the experiment using Asymmetric generator is useful for robustness evaluation. However, the experiment is based on the learned response model. In addition, expert evaluation supports the reduction of information leakage, but it doesn't validate the measurement reactivity of actual respondents.
Accordingly, I will keep my rating.
Clarification of Direction Comparison, Asymmetric Comparison and Expert Study
W1. Direct evaluation of trait estimates
Thank you for pointing out the direction for comparing trait estimates.
Reason for Using GRM/EAP. Our previous comparison standardized the item budget and scoring procedure. This controlled comparison evaluates measurement information retained by selection. We now additionally evaluate each method's original trait output.
Model Input vs True Trait. We conducted an additional experiment to directly compare the models' trait estimates with the full-questionnaire true trait given by the questionnaire rules. For each CAT method's endpoint estimate and the true full-questionnaire trait , we compare
{\sqrt{\sum_i(x_i-\bar x)^2\sum_i(y_i-\bar y)^2}}, \qquad \rho_m=\operatorname{corr}_{\mathrm{Pearson}}\!\left(R(x),R(y)\right),$$ where $R$ denotes ranks to eliminate the impact of units. Detailed results are shown in the table below, where only the best baseline method is shown. | Dataset | Method | Pearson r ↑ | Spearman ρ ↑ | | --- | --- | ---: | ---: | | DASS-42 | PsyCAT | **0.9791 ± 0.0005** | **0.9792 ± 0.0013** | | DASS-42 | UATS | 0.9617 ± 0.0016 | 0.9648 ± 0.0017 | | Xinxiang-23 | PsyCAT | **0.9778 ± 0.0031** | **0.9626 ± 0.0084** | | Xinxiang-23 | EAP-MFI | 0.9039 ± 0.0300 | 0.8864 ± 0.0430 | | Corona-23 | PsyCAT | **0.9885 ± 0.0013** | **0.9891 ± 0.0008** | | Corona-23 | UATS | 0.9664 ± 0.0022 | 0.9685 ± 0.0027 | We can see that, with high Pearson and Spearman values, PsyCAT also outperforms all baseline methods, showing the efficiency of our method. **External independent measurement.** We selected **two additional datasets** to extend the DASS and PROMIS analyses in the submission. In the Transdiagnostic Connectome Project (TCP) [1], participants reported their symptoms on the DASS, while **clinicians assessed the same participants’ depression severity using the Montgomery–Åsberg Depression Rating Scale (MADRS) as an independent measurement**. In the PROMIS Pediatric Measure Evaluation [2], children reported their own physical activity, and **parents separately rated the same child’s activity as an independent measurement**. All the models and experiments remain the same as in the submitted version. For each fit, let $\hat{\theta}_{m,i}$ denote method $m$’s trait estimate for participant $i$ and $y_i$ the corresponding external rating. We computed Spearman’s rank correlation across participants and the paired difference between methods: $$\rho_m = \operatorname{corr}_{\mathrm{Spearman}}\!\left((\hat{\theta}_{m,i})_i, (y_i)_i\right), \qquad \Delta\rho = \rho_{\mathrm{PsyCAT}} - \rho_{\mathrm{EAP\text{-}MFI}}.$$ Detailed results are shown in the table below. | Dataset and target trait | Item budget | PsyCAT ρ | Δρ | |---|---:|---:|---:| | TCP: depression severity [1] | 15 | 0.7057 ± 0.0077 | +0.0162 ± 0.0067 | | PROMIS: physical activity [2] | 20 | 0.7268 ± 0.0497 | +0.0027 ± 0.0019 | As shown in the table above, PsyCAT achieves high trait estimation accuracy when evaluated against the independent measurement, and it outperforms the baseline. **W2. Model dependence and measurement reactivity** **Asymmetric generator.** Parameters learned from real responses capture empirical patterns. Stochastic sampling, trait/style interventions and nonuniform threshold perturbations test sensitivity to generator–estimator matching. We respectfully consider this a reasonable controlled robustness evaluation within its model-based scope. **Expert evaluation.** Information leakage concerns whether questions reveal the system's assessment; measurement reactivity concerns measurement-induced changes in behavior or self-reports [3]. Disclosed assessment cues may influence subsequent reporting through demand characteristics [4]. MERIT uses expert judgment to identify reactivity risks, including disclosure of measured outcomes [3]; clinical expert review also supports auditing simulated mental-health interactions [5]. Given that **our primary contribution is algorithmic**, the blinded expert task therefore **evaluates a plausible informational pathway to reactivity, with the estimator and item budget controlled.** We would be grateful if you could consider the rating again in the light of the evidence above. **References** [1] [Chopra et al., *Sci. Data* (2025)](https://doi.org/10.1038/s41597-025-04895-z). [2] [Forrest, *PROMIS Pediatric Measure Evaluation* (2021)](https://doi.org/10.7910/DVN/QOO7QX). [3] [French et al., *J. Clin. Epidemiol.* (2021)](https://doi.org/10.1016/j.jclinepi.2021.06.028). [4] [Orne, *Am. Psychol.* (1962)](https://doi.org/10.1037/h0043424). [5] [Weilnhammer et al., *Nat. Med.* (2026)](https://doi.org/10.1038/s41591-026-04577-2).
Clarification of Full-Questionnaire Direct Comparison, Trait–Style Separation, and Reactivity Evidence
We thank the reviewer for highlighting the distinction between response prediction, assessment accuracy, and measurement reactivity.
W1. Comparison with full-questionnaire scores
Direct comparison. We agree that a direct comparison better demonstrates our method's efficacy. We reconstructed each domain's full score from observed scores on administered items and expected scores on unadministered items:
S=∑j∈Ayj+∑j∈/AE[Yj∣H20],Sfull=∑jyj,
where sums run over a domain's items, A is its administered subset, and H20 contains the 20 administered item–response pairs. For trait estimates, we applied the same frozen GRM/EAP readout to each method's selected responses and to the complete questionnaire:
θ=Eref[θ∣H20],θfull=Eref[θ∣Y1:42].
This places all methods on a common measurement scale for comparing the information retained by item selection. Unadministered responses were used only to construct the references. Additionally, grade accuracy measures agreement with full-questionnaire five-level severity labels, averaged equally over depression, anxiety, and stress.
All results use the same frozen models and five seeds as the main experiments. PsyCAT attains the lowest error on both measures. We will add this comparison to the revision.
Why we reported held-out prediction metrics. The reason why we only report the held-out set prediction metrics is that the GRM connects latent traits to observable responses. In our response-style extension
Pr(Yj>k∣θ,s)=σ(ajθdj−τjk(s)),τjk(s)=τˉj+eλjs(τjk−τˉj),
where θdj is the trait measured by item j, s is the response style, and τjk(s) is the style-dependent threshold. Given history HT, structural predictions integrate over the inferred joint posterior:
Pstruct(Yj=c∣HT)=∫Pr(Yj=c∣θ,s)q(θ,s∣HT)dθds.
Held-out prediction thus tests the observable implications of the inferred trait–style posterior. ACC assesses the most probable category, AUC category-wise discrimination across respondents, and RPS the full predictive distribution while respecting category order. As PsyCAT also applies posterior and probability calibration, these metrics evaluate the complete prediction pipeline. We agree they complement, rather than replace, the direct comparison above.
W2. Scope of trait–style separation and reactivity evidence
Trait–style separation. The generators use item parameters calibrated on real DASS training responses, with randomly sampled traits and stochastic response sampling, giving an real-world grounded setting for testing controlled trait–style changes and robustness to specified misspecification while keeping it not directly affected by learned models.
In the model above, θ locates a respondent on the symptom-severity axis, while s controls the propensity for central versus extreme categories. And positive s expands threshold spacing and favors central categories while negative s contracts it and favors extreme ones. To test departures from this uniform scaling, our asymmetric generator lets different threshold gaps within an item respond to style at different rates, introducing structural misspecification while retaining the calibrated item parameters, the GRM's logistic link, and the specified latent structure. These experiments thus test change recovery beyond the exact uniform-scaling model.
Measurement reactivity. We agree that expert predictions differ from respondents' experience, given the gap in domain knowledge, yet expert judgment is used to assess psychological constructs and communication quality [1,2].
Reference
[1] Kumar et al. When large language models are reliable for judging empathic communication. Nature Machine Intelligence (2026).
[2] Tu et al. Towards conversational diagnostic artificial intelligence. Nature (2025).