acceptodds
Under review as a conference paper at ICLR 2027

PsyCAT: Robust Posterior Mixtures for Adaptive Self-Report Assessment

Abstract

Computerized adaptive testing (CAT) can shorten self-report questionnaires, reducing respondent burden while preserving assessment efficiency. Yet self-report CAT faces two challenges. First, self-report responses have no objectively correct answer, and differences in category interpretation and preferences for extreme or intermediate options can confound the latent trait with response style. Second, maximum-information selection can repeatedly probe one trait, revealing the system's provisional assessment and potentially inducing measurement reactivity. PsyCAT is proposed to systematically address these challenges. Response style is explicit modelled in proposed method in an ordinal measurement model and derives how uncertainty in style increases trait-estimation risk. A Gaussian mixture characterizes the joint posterior over latent traits and response style, separating uncertainty within components from disagreement across components. Shared-attention post-calibration adjusts these posteriors using observed answers, and minimax aggregation produces trait estimates. PsyCAT further controls information leakage by penalizing dependence between next-item selection and alternative response profiles. Across several published questionnaire datasets, PsyCAT achieved the lowest mean endpoint ranked probability score among the evaluated methods. Under controlled latent interventions, PsyCAT tracked response-style changes with little average trait drift in the style-only condition and tracked both variables in the joint condition. In a two-expert study, its selection policy is proven to reduce information leakage.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Computerized adaptive testing, Psychometrics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nJdLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PsyCAT for self-report computerized adaptive testing. It combines explicit response-style modelling, Gaussian posterior mixtures, shared-attention post-calibration, minimax aggregation, and information-regularized item selection. Evaluation includes held-out response prediction on six questionnaire banks, controlled latent-change experiments, and an expert study of information leakage.

Critical strengths
  • The paper addresses response-style variation and information leakage in self-report CAT through an integrated estimation and selection framework.
  • PsyCAT achieves the lowest mean endpoint RPS among the evaluated methods across six questionnaire banks, providing consistent evidence of improved response prediction.
Critical weaknesses
  • The stated objective is to minimize trait-estimation error, but the main real-data evaluation focuses on held-out response prediction. No direct comparison between CAT trait estimates and full-questionnaire scores or estimates is reported. This limits the evidence for improved assessment accuracy.
  • The evidence for trait–style separation remains model-dependent. The controlled latent-change experiments use generators derived from fitted response models, while the style model captures threshold-spacing preferences. The expert study evaluates sequence inferability rather than measurement reactivity.
Questions for authors

See the weakness section.

3 comments
AuthorsOpenReview ↗

Clarification of Full-Questionnaire Direct Comparison, Trait–Style Separation, and Reactivity Evidence

We thank the reviewer for highlighting the distinction between response prediction, assessment accuracy, and measurement reactivity.

W1. Comparison with full-questionnaire scores

Direct comparison. We agree that a direct comparison better demonstrates our method's efficacy. We reconstructed each domain's full score from observed scores on administered items and expected scores on unadministered items:

S^=∑j∈Ayj+∑j∉AE[Yj∣H20],Sfull=∑jyj,\widehat S=\sum_{j\in A}y_j+\sum_{j\notin A}\mathbb E[Y_j\mid H_{20}],\qquad S_{\mathrm{full}}=\sum_j y_j,

where sums run over a domain's items, AA is its administered subset, and H20H_{20} contains the 20 administered item–response pairs. For trait estimates, we applied the same frozen GRM/EAP readout to each method's selected responses and to the complete questionnaire:

θ^=Eref[θ∣H20],θ^full=Eref[θ∣Y1:42].\widehat\theta=\mathbb E_{\mathrm{ref}}[\theta\mid H_{20}],\qquad\widehat\theta_{\mathrm{full}}=\mathbb E_{\mathrm{ref}}[\theta\mid Y_{1:42}].

This places all methods on a common measurement scale for comparing the information retained by item selection. Unadministered responses were used only to construct the references. Additionally, grade accuracy measures agreement with full-questionnaire five-level severity labels, averaged equally over depression, anxiety, and stress.

Method (DASS)Full-score RMSE ↓Trait-estimate RMSE ↓Grade accuracy ↑ (%)
PsyCAT2.676 ± 0.0490.2249 ± 0.006275.80 ± 0.98
EAP-MFI2.852 ± 0.0910.2488 ± 0.008975.11 ± 1.14
Transformer-U2.705 ± 0.1060.2593 ± 0.005574.64 ± 1.31
NCAT2.760 ± 0.0360.2522 ± 0.007675.11 ± 1.14
UATS2.727 ± 0.0710.2410 ± 0.011275.08 ± 0.99
RECAT2.820 ± 0.0880.2447 ± 0.009475.11 ± 1.14

All results use the same frozen models and five seeds as the main experiments. PsyCAT attains the lowest error on both measures. We will add this comparison to the revision.

Why we reported held-out prediction metrics. The reason why we only report the held-out set prediction metrics is that the GRM connects latent traits to observable responses. In our response-style extension

Pr⁡(Yj>k∣θ,s)=σ ⁣(ajθdj−τjk(s)),τjk(s)=τˉj+eλjs(τjk−τˉj),\Pr(Y_j>k\mid\theta,s)=\sigma\!\left(a_j\theta_{d_j}-\tau_{jk}(s)\right),\qquad\tau_{jk}(s)=\bar\tau_j+e^{\lambda_j s}(\tau_{jk}-\bar\tau_j),

where θdj\theta_{d_j} is the trait measured by item jj, ss is the response style, and τjk(s)\tau_{jk}(s) is the style-dependent threshold. Given history HTH_T, structural predictions integrate over the inferred joint posterior:

Pstruct(Yj=c∣HT)=∫Pr⁡(Yj=c∣θ,s) q(θ,s∣HT) dθ ds.P_{\mathrm{struct}}(Y_j=c\mid H_T)=\int\Pr(Y_j=c\mid\theta,s)\,q(\theta,s\mid H_T)\,d\theta\,ds.

Held-out prediction thus tests the observable implications of the inferred trait–style posterior. ACC assesses the most probable category, AUC category-wise discrimination across respondents, and RPS the full predictive distribution while respecting category order. As PsyCAT also applies posterior and probability calibration, these metrics evaluate the complete prediction pipeline. We agree they complement, rather than replace, the direct comparison above.

W2. Scope of trait–style separation and reactivity evidence

Trait–style separation. The generators use item parameters calibrated on real DASS training responses, with randomly sampled traits and stochastic response sampling, giving an real-world grounded setting for testing controlled trait–style changes and robustness to specified misspecification while keeping it not directly affected by learned models.

In the model above, θ\theta locates a respondent on the symptom-severity axis, while ss controls the propensity for central versus extreme categories. And positive ss expands threshold spacing and favors central categories while negative ss contracts it and favors extreme ones. To test departures from this uniform scaling, our asymmetric generator lets different threshold gaps within an item respond to style at different rates, introducing structural misspecification while retaining the calibrated item parameters, the GRM's logistic link, and the specified latent structure. These experiments thus test change recovery beyond the exact uniform-scaling model.

Measurement reactivity. We agree that expert predictions differ from respondents' experience, given the gap in domain knowledge, yet expert judgment is used to assess psychological constructs and communication quality [1,2].

Reference

[1] Kumar et al. When large language models are reliable for judging empathic communication. Nature Machine Intelligence (2026).

[2] Tu et al. Towards conversational diagnostic artificial intelligence. Nature (2025).

Thank you for your additional experimental results and comments.

  • W1: I think that the comparison with the full-questionnaire results are useful and it solves most of my concern. However, I think the common GRM/EAP is used to compare the trait estimated value and it doesn't directly evaluate the effectiveness of original trait estimation method.
  • W2: I think that the experiment using Asymmetric generator is useful for robustness evaluation. However, the experiment is based on the learned response model. In addition, expert evaluation supports the reduction of information leakage, but it doesn't validate the measurement reactivity of actual respondents.

Accordingly, I will keep my rating.

AuthorsOpenReview ↗

Clarification of Direction Comparison, Asymmetric Comparison and Expert Study

W1. Direct evaluation of trait estimates

Thank you for pointing out the direction for comparing trait estimates.

Reason for Using GRM/EAP. Our previous comparison standardized the item budget and scoring procedure. This controlled comparison evaluates measurement information retained by selection. We now additionally evaluate each method's original trait output.

Model Input vs True Trait. We conducted an additional experiment to directly compare the models' trait estimates with the full-questionnaire true trait given by the questionnaire rules. For each CAT method's endpoint estimate xix_i and the true full-questionnaire trait yiyi, we compare

{\sqrt{\sum_i(x_i-\bar x)^2\sum_i(y_i-\bar y)^2}}, \qquad \rho_m=\operatorname{corr}_{\mathrm{Pearson}}\!\left(R(x),R(y)\right),$$ where $R$ denotes ranks to eliminate the impact of units. Detailed results are shown in the table below, where only the best baseline method is shown. | Dataset | Method | Pearson r ↑ | Spearman ρ ↑ | | --- | --- | ---: | ---: | | DASS-42 | PsyCAT | **0.9791 ± 0.0005** | **0.9792 ± 0.0013** | | DASS-42 | UATS | 0.9617 ± 0.0016 | 0.9648 ± 0.0017 | | Xinxiang-23 | PsyCAT | **0.9778 ± 0.0031** | **0.9626 ± 0.0084** | | Xinxiang-23 | EAP-MFI | 0.9039 ± 0.0300 | 0.8864 ± 0.0430 | | Corona-23 | PsyCAT | **0.9885 ± 0.0013** | **0.9891 ± 0.0008** | | Corona-23 | UATS | 0.9664 ± 0.0022 | 0.9685 ± 0.0027 | We can see that, with high Pearson and Spearman values, PsyCAT also outperforms all baseline methods, showing the efficiency of our method. **External independent measurement.** We selected **two additional datasets** to extend the DASS and PROMIS analyses in the submission. In the Transdiagnostic Connectome Project (TCP) [1], participants reported their symptoms on the DASS, while **clinicians assessed the same participants’ depression severity using the Montgomery–Åsberg Depression Rating Scale (MADRS) as an independent measurement**. In the PROMIS Pediatric Measure Evaluation [2], children reported their own physical activity, and **parents separately rated the same child’s activity as an independent measurement**. All the models and experiments remain the same as in the submitted version. For each fit, let $\hat{\theta}_{m,i}$ denote method $m$’s trait estimate for participant $i$ and $y_i$ the corresponding external rating. We computed Spearman’s rank correlation across participants and the paired difference between methods: $$\rho_m = \operatorname{corr}_{\mathrm{Spearman}}\!\left((\hat{\theta}_{m,i})_i, (y_i)_i\right), \qquad \Delta\rho = \rho_{\mathrm{PsyCAT}} - \rho_{\mathrm{EAP\text{-}MFI}}.$$ Detailed results are shown in the table below. | Dataset and target trait | Item budget | PsyCAT ρ | Δρ | |---|---:|---:|---:| | TCP: depression severity [1] | 15 | 0.7057 ± 0.0077 | +0.0162 ± 0.0067 | | PROMIS: physical activity [2] | 20 | 0.7268 ± 0.0497 | +0.0027 ± 0.0019 | As shown in the table above, PsyCAT achieves high trait estimation accuracy when evaluated against the independent measurement, and it outperforms the baseline. **W2. Model dependence and measurement reactivity** **Asymmetric generator.** Parameters learned from real responses capture empirical patterns. Stochastic sampling, trait/style interventions and nonuniform threshold perturbations test sensitivity to generator–estimator matching. We respectfully consider this a reasonable controlled robustness evaluation within its model-based scope. **Expert evaluation.** Information leakage concerns whether questions reveal the system's assessment; measurement reactivity concerns measurement-induced changes in behavior or self-reports [3]. Disclosed assessment cues may influence subsequent reporting through demand characteristics [4]. MERIT uses expert judgment to identify reactivity risks, including disclosure of measured outcomes [3]; clinical expert review also supports auditing simulated mental-health interactions [5]. Given that **our primary contribution is algorithmic**, the blinded expert task therefore **evaluates a plausible informational pathway to reactivity, with the estimator and item budget controlled.** We would be grateful if you could consider the rating again in the light of the evidence above. **References** [1] [Chopra et al., *Sci. Data* (2025)](https://doi.org/10.1038/s41597-025-04895-z). [2] [Forrest, *PROMIS Pediatric Measure Evaluation* (2021)](https://doi.org/10.7910/DVN/QOO7QX). [3] [French et al., *J. Clin. Epidemiol.* (2021)](https://doi.org/10.1016/j.jclinepi.2021.06.028). [4] [Orne, *Am. Psychol.* (1962)](https://doi.org/10.1037/h0043424). [5] [Weilnhammer et al., *Nat. Med.* (2026)](https://doi.org/10.1038/s41591-026-04577-2).
Reviewer_GFC6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PsyCAT, a framework for adaptive self-report assessment to model latent traits through an ordinal response model. To account for calibration uncertainty, PsyCAT forms a mixture of posteriors from bootstrap fitted models, applies a shared-attention post-calibration module, and reports traits using a minimax aggregation over ensemble members. The authors evaluate PsyCAT on six questionnaire datasets, where they reported improved endpoint ranked probability scores relative to several adaptive testing baselines.

Critical strengths
  1. The paper addresses a meaningful problem in self-report CAT: response style variation can be confounded with the trait being measured. The explicit joint trait style model as well as the derivation of excess risk from unobserved style give the work a clear modeling rationale.

  2. The evaluation spans predictive performance, controlled latent changes, and observer inference from item sequences. The paper also distinguishes the mathematical profile information quantity from the empirical expert inference endpoint. Moreover, the paper acknowledges that it does not test behavioral reactivity during live administration.

Critical weaknesses
  1. The central claim that selection reduces information leakage is supported by a narrow expert study: only two experts evaluated sequences from one questionnaire bank, and the appendix notes that case identifiers were unavailable, so pairing or reuse could not be assessed. This limits the strength and generality of the empirical evidence.

  2. The latent trait-style separation evidence is based on controlled interventions and a shared frozen decoder, rather than independent external trait measurements or prospective self-report data.

Questions for authors
  1. Can you report the expert-study test results and effect sizes directly, and clarify whether any sequences were repeated or shared across experts?

  2. How sensitive are the trait–style separation results to misspecification beyond the asymmetric threshold gap generators tested here, especially to location-shift response styles? Do you have evidence against an external measure of the target trait?

1 comment
AuthorsOpenReview ↗

Clarification of New Expert Study, Location-Shift Experiment, and External Evidence

We thank the reviewer for the constructive assessment and respond to each point below.

W1/Q1. Expert-study design, pairing, and scope

Original study. We'd like to first clarify the expert study in submission version first. Both experts hold PhDs in psychology and have over 1,600 (Expert A) and 1,400 (Expert B) counseling hours. And the sequenced reviewed are not paired. The reason why expert reviewed different test is to cover more samples given the total sample size is fairly large and very limited psychological professionals. Expert A rated 81 PsyCAT and 69 PsyCAT-MFI sequences while Expert B rated 69 and 81 to offset the workload difference impact. Each sequence showed only the text and order of 15 selected Xinxiang-23 items with their responses remained concealed. The expert nominated one domain (depression, anxiety, or insomnia) and predicted its severity on the study-defined 0–3 scale. We scored exact four-level and binary (0–1 vs. 2–3) agreement. Binary accuracy was 66.67% for PsyCAT and 86.00% for PsyCAT-MFI, a difference of −19.33 percentage points (two-sided expert-stratified exact test, p = 0.000136).

New participant-matched study. We agree that a paired design is a more direct test of the selection strategy, and we made an addtional study to show that. The same experts rated sequences from 100 new participants (no overlap with the original study) under the same presentation and judgment protocol. For participants 1–50, Expert A rated the PsyCAT sequence and Expert B the PsyCAT-MFI sequence and for participants 51–100, the reverse. Binary accuracy was 41% for PsyCAT versus 65% for PsyCAT-MFI, a paired difference of −24 percentage points (95% participant-level bootstrap CI, stratified by expert assignment: −36 to −12). A participant-stratified conditional logistic regression adjusting for expert identity gave the same conclusion (OR = 0.280, 95% CI 0.133–0.588, p = 0.000770). This confirms the original direction while controlling for participant differences.

Role of expert judgment. We agree that expert predictions are not equivalent to respondents' own experience and that two experts is a limitation. Expert assessment is nevertheless an established evaluation tool for empathic communication and clinical dialogue [1,2]. Our central contribution is algorithmic, a method that reduces the subjectivity of self-assessment, and we do not claim clinical validation. We will state this scope in the revision and leave a larger multi-expert study to future work.

W2/Q2. Location-shift robustness and evidence boundary

New location-shift experiment. Following the reviewer's suggestion, we ran a stress test on DASS, Xinxiang, and Corona same settings to analyze the sensitivity of trait–style separation results. We interpret the location-shift as a tendency to report systematically higher or lower ratings despite the same triat. And we model the responses generation by

P(Yij>k)=σ[ajθi,dj−τjk(si)−ajδiwj],P(Y_{ij}>k)=\sigma[a_j\theta_{i,d_j}-\tau_{jk}(s_i)-a_j\delta_i w_j],

where δi\delta_i is a respondent-specific threshold-location shift and wj=ρ+(1−ρ)vjw_j=\rho+(1-\rho)v_j, with vj∈{−1,+1}v_j\in\{-1,+1\}, sets item-specific shift magnitudes. We varied ρ∈{0,0.25,0.5,0.75,1}\rho\in\{0,0.25,0.5,0.75,1\} and ∣δi∣∈{0.5,1}|\delta_i|\in\{0.5,1\}, with positive and negative shifts balanced across respondents. All other settings and frozen models are identical to the submission. The table reports RMSE of estimated trait change (mean ± sample SD over five seeds; lower is better).

BankPsyCATOrdinary GRM
DASS0.717 ± 0.0140.850 ± 0.027
Xinxiang0.733 ± 0.0240.870 ± 0.036
Corona0.742 ± 0.0380.905 ± 0.063

PsyCAT has lower error than the ordinary GRM on all three banks. This supports relative robustness to the specified location-shift misspecification.

External Evidence. For external evidence to show the PsyCAT efficacy on measure target triat we directy compare the severity-grade accuracy (%) of PsyCAT diagnosis vs full-questionnaire labels which can be considered as gold-standard comparison. Values are means ± sample SD over five seeds, with equal weighting across domains averaged equally over depression, anxiety, and stress.

DatasetPsyCATRECAT/GRMTransformer-U
DASS75.80 ± 0.9875.11 ± 1.1474.64 ± 1.31
Xinxiang95.55 ± 0.1395.48 ± 0.1295.42 ± 0.29
Corona86.72 ± 1.0086.11 ± 0.9886.40 ± 1.04

References

[1] Kumar et al. When large language models are reliable for judging empathic communication. Nature Machine Intelligence, 2026.

[2] Tu et al. Towards conversational diagnostic artificial intelligence. Nature, 2025.

Reviewer_UzjDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PsyCAT, a computerized adaptive testing (CAT) method for self-report questionnaires. It extends the graded response model with a response-style variable that stretches or compresses the category thresholds. Authors then represent calibration uncertainty as a Gaussian mixture over the original fit and other bootstrap refits. On held-out response prediction over six questionnaires, PsyCAT has the lowest mean endpoint RPS in all six.

Critical strengths
  • The problem is well motivated and is a real life problem.
  • The prediction benchmarks are broad and carefully reported.
Critical weaknesses
  1. In the experiments, the improvement looks marginal compared to other baselines. There is no standard deviation reported, so it's hard to interpret if the improvement is significant.
  2. PsyCAT has multiple components (style variable, bootstrap mixture, Transformer correction, minimax aggregation, regularized selection, etc) and none of them is ablated. As a result, it's not clear what produces the gains in PsyCAT.
  3. The theorems introduced in section 3 are not very clear to me. Authors could better explain how they lead to the methods introduced in the section 4. The logical connections between theorems is not well explained and motivated.
Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Reliable Gains, Ablation Study and Methodological Clarification

We thank the reviewer for the constructive feedback and address each point below.

W1. Standard deviations and marginal improvements.

Standard deviations. Appendix C (Tables 6–9, pp. 23–25) reports mean ± sample SD over five seeds (2026–2030), as referenced in the caption of Table 2.

Improvement. Statistical testing comparing our model with the best model on each dataset shows that the improvement is significant.

DatasetAdjusted p-value
DASS0.000030
Xinxiang0.00135
Corona0.00123

All results were confirmed by a rigorous rerun. These results will be made more visible in the future version.

W2. Component comparisons.

Thank you for pointing out the importance of identifying the sources of the gains. Please find the ablation study results below.

DatasetFullNo styleNo bootstrapNo attentionEqual weightsNo info. reg.
DASS-42 (20)0.11978±0.002050.11999±0.002310.11984±0.001930.12092±0.001560.12001±0.001820.11995±0.00195
Xinxiang-23 (15)0.04399±0.001030.04419±0.001190.04460±0.000900.04451±0.001060.04402±0.001150.04404±0.00108
Corona-23 (15)0.08857±0.004200.08887±0.004210.08871±0.004230.09913±0.004190.08877±0.004180.08885±0.00423

We respectfully believe that each component addresses a realistic problem including style (heterogeneous category use), bootstrap (calibration instability), attention (local independence), minimax (predefined weights), and information regularization (profile leakage). The ablation study results will be added.

W3. Motivation of Section 3.

Thank you for your suggestion. Please a pseudocode is presented below for better logic connection.

H ← ∅
for t = 0, …, T:
  q_b ← JointPosterior(H; Φ_b)               # Eqs. 2–3
  q̃_b, P̃_b ← ContextCorrection(q_b, H; ψ)    # Eqs. 11–12
  μ_b, v_b ← E_q̃_b[θ], tr Cov_q̃_b(θ)         # Eq. 10
  θ̂, w* ← MinimaxReport(μ, v)                # Eqs. 13–14
  if t = T: return θ̂, Σ_b w_b* P̃_b
  c ← ProfileRisk(H, Q \ items(H))           # Eq. 15
  π ← PolicyUpdate(c, κ)                     # Eqs. 16–17
  j ∼ π_0; H ← H ∪ {(j, Answer(j))}

The pseudocode will be added in the future version. Please also find a detailed walkthrough below.

The reasoning leading to Proposition 1 is that define response style (Eqs. 1–3) → quantify the cost of unknown style (Eqs. 4–6) → explain this cost through variation in trait reports (Eq. 7). This links style modeling to risk-based reporting and selection (Eqs. 13, 15). For fixed answers and calibration, Eq. 7 gives a direct measurement of the Style Uncertainty Penalty, i.e., the difference between knowing the style and not conditioning on it. The trace form uses conditional-report moments without constructing a full covariance matrix, summing contributions across trait dimensions under Eq. 1’s squared-error criterion.

Motivation for the GMM. Finite data can destabilize calibration, especially for underrepresented response patterns. With no gold standard separating trait from style, identical answers may support different interpretations across training samples. The flow bootstrap resampling → multiple calibrations → joint posteriors per respondent → Gaussian moments → GMM makes this sensitivity explicit. The mixture retains both within-calibration uncertainty and cross-calibration disagreement, separating limited personal evidence from measurement-parameter variation and reducing reliance on a single calibration.

Shared attention. All refits assume local independence, whereas questionnaire content may alter self-report comparison standards. Attention over observed item–answer pairs and candidate items tilts each joint posterior (Eq. 11) and adjusts category probabilities (Eq. 12); the corrected moments enter the same risk calculation.

Proposition 2 (minimax aggregation). It links calibration disagreement → risk-based weighting (Eq. 13) → robust reporting (Eq. 14). The report minimizes the largest expected squared error across corrected members and tractably reduces to the weighted mean of member estimates under optimized least-favorable mixture weights. The weights follow from the measurement objective, reflecting posterior uncertainty and location disagreement. Eq. 15 reuses this rule for each hypothetical answer.

Risk- and leakage-aware selection. Eq. 15 scores each candidate by its expected remaining risk, recomputing the minimax report for each possible answer, averaging within members and maximizing across them, so that selection and reporting target the same error. Repeated questions on one domain may disclose the suspected profile; Eq. 16 trades off risk against this disclosure (mutual information between actual and reference profiles), and Eq. 17 yields the randomized policy. The observed answer updates the history, closing the inference–reporting–selection loop.

Thanks you for the detailed reply and additional experiments. However, I think the paper needs significant rewrite to improve clarity and presentation. For example, I think the ablation study needs to be discussed in the main text due to the various introduced components. The theory part, even with the additional explanation, still feels confusing to me. Thus, my score remains at 2.

AuthorsOpenReview ↗

Clarification on Method Rationale and Ablation Study

Thank you for all your suggestions. And I do believe they help to make this paper better. With all your points and concerns in mind, we would like to state again the motivation, the flow, and the logic behind not reporting an ablation study.

Theoretical rationale. PsyCAT addresses two long-standing problems in self-report assessment.

  1. Subjective responses. Respondents at the same trait level may use rating categories differently. These differences can distort trait scores [1].
  2. Testing reactivity. Repeated questions about depression, for example, may reveal the test’s provisional assessment. Respondents can pick up on such cues and adjust how they answer [2].

In the method section, we generally follow the logic below.

  1. Start with the inspiring problem.
  2. Formulate the problem and the new method scientifically.
  3. Identify the challenge in the formulation.
  4. Propose an algorithmic enhancement.

For each part, the flow is exactly as follows:

  • Model response style. Differences in category use (1) → extend the graded response model (GRM) to estimate trait and style jointly (2) → quantify the estimation risk due to unobserved style (Proposition 1, 3) → use a Gaussian mixture model (GMM) to retain alternative calibrations of the trait–style relationship (4).
  • Represent uncertainty. Disagreement across bootstrap calibrations (1) → represent each calibration’s posterior as a component of the mixture. When responses depend on earlier items even after conditioning on trait and style, the GRM’s conditional independence assumption breaks down (2&3) → use shared attention over prior items and answers to correct the posteriors (4).
  • Report and select. Corrected posteriors can yield different trait estimates (1) → choose the estimate that minimizes the largest squared-error risk across calibrations (Proposition 2, 2&3) → select the next item using the same risk criterion, with a penalty for revealing the provisional profile (4).

We will clarify this flow in the revision. We acknowledge that the readability of the current version can be improved. However, some of the difficulty is inherent in the technical nature of statistics-informed methods.

Ablation study. We initially evaluated the complete procedure because each component addresses a specific measurement problem. Removing the mixture, for example, leaves calibration disagreement unaddressed. Theory-driven CAT studies such as BECAT [3] and CCAT [4] informed our rationale for not reporting an ablation study. Their main evaluations focus on ability estimation and ranking consistency, respectively. Your suggestion has surely helped us realize the value of the reduced models as controls. And we will revise this in the next version.

In light of these comparisons and the statistical rationale connecting trait estimation to item selection, we respectfully ask you to reconsider your rating.

References

  1. Laajaj, R., et al. (2019). Challenges to capture the big five personality traits in non-WEIRD populations. Science Advances, 5, eaaw5226.
  2. Orne, M. T. (1962). On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American Psychologist, 17, 776–783.
  3. Zhuang, Y., et al. (2023). A Bounded Ability Estimation for Computerized Adaptive Testing. NeurIPS.
  4. Liu, Z., et al. (2024). Computerized Adaptive Testing via Collaborative Ranking. NeurIPS.

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

The discussion period ends on November 18.

Thanks,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.