- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for measuring behavioral propensities in AI systems, distinguishing them from capabilities. The authors introduce a two-sided logistic IRT model (2×2PL) in which task success depends on whether a model's propensity falls within an acceptable range. They use LLM-annotated task requirements and synthetic datasets to estimate four propensities across several language models. The results show that estimated propensities generally respond to prompting and provide some additional predictive value when combined with capability measures, particularly on held-out benchmarks.
Critical strengths
-
The framework is sound. The distinction between capability and behavioral tendency is a useful one: a model may be fully capable of a task and still fail because of how it tends to behave. The paper gives a formal way to study that gap.
-
The evaluation is fairly thorough. It covers several models and propensity dimensions and includes held-out benchmarks. The results suggest that propensity information can complement capability measures when predicting model performance.
Critical weaknesses
-
My first concern is methodological novelty. Non-monotonic item response models, including ideal-point and unfolding models, are well established in psychometrics. As far as I can tell, 2×2PL is a variant of these that adds an acceptable propensity interval. That adaptation could be worthwhile, but the paper does not directly compare it against the existing non-monotonic IRT models. As a result, I could not judge whether the new formulation improves parameter estimation or prediction, or only re-parameterizes a known model.
-
My biggest worry is construct validity of the estimated propensities. Most of the evidence comes from synthetic tasks, prompted behavior changes, and annotations generated by an LLM. Seeing that a score shifts when the system prompt tells the model to behave differently does not convince me that the score measures a stable tendency. It may simply track compliance with that prompt on that task. I also found little independent validation of the annotations or of the resulting measurements.
Questions for authors
-
How does 2×2PL compare with established ideal-point or unfolding IRT models in terms of parameter recovery, model fit, and predictive performance?Even one direct comparison would go a long way.
-
What evidence supports the construct validity of the estimated propensities? In other words, how can the authors distinguish stable behavioral tendencies from responses induced by system prompts?
-
How reliable are the propensity demand intervals annotated by GPT-4.1? Can the authors provide independent human validation? Having humans check a sample would help. Is it possible that a comparison with simpler behavioral measures do not require fitting the proposed IRT model?
2 comments
Thank you for the detailed response and the new analyses.
The comparison with HCM is helpful, but it also confirms my concern about novelty. The functional form is essentially an existing unfolding model, and the gains over HCM are modest with equal parameter recovery. The main contribution is annotating the interval bounds, which makes population-free estimation possible. The paper mainly shows that prompt-induced shifts transfer, and it cannot establish stable default propensities.
The GPT-4.1 intervals also remain without human validation. I also could not find these new analyses of “W2.3” in the current PDF. The paper's presentation can be improved with careful revisions.
Rebuttal Response to h2ps
We thank the reviewer for recognising the usefulness of separating capabilities from propensities and the breadth of our evaluation. We now address the points raised by h2ps (all new analyses use the released data, with no new model runs):
W1. Limited novelty with respect to ideal-point and unfolding IRT: we agree that the 2x2PL belongs to the family of unfolding models: per item, Eq. 5 is a peak-normalised hyperbolic cosine model (HCM; Andrich & Luo, 1993; Johnson & Junker, 2003), and we will state this explicitly. Our contribution does not lie in the functional form, but in (i) its parametrisation by the lower and upper bounds of a tolerated interval, which can be annotated per item, so item parameters come from annotation rather than from a population fit; (ii) the reading of capabilities and propensities as the one- and two-sided cases of the same interval; and (iii) the estimation of a single subject's level without a population of respondents. The empirical comparison with HCM is given in Q1.
W2. Construct validity: we address the three alternative explanations raised by the reviewer.
Q1. Comparison with ideal-point / unfolding models: we compared 2x2PL with HCM under identical item parameters (fixed by the annotations), over 383 (dimension, model, condition) cells:
Q2. Stable tendencies vs prompt-induced responses: see W2.1. Our evidence supports θ as a common scale on which propensity shifts are measured and transfer to natural tasks. Establishing stable between-model differences in default propensities requires larger item banks than ours, which we will state as a limitation.
Q3. Reliability of GPT-4.1 intervals, and simpler behavioural measures: human validation of the intervals is not reported, and we will state it as a limitation; W2.3 gives the evidence that they carry item-specific content. Against the simplest behavioural measure that needs no IRT fit, the aggregate failure rate:
We hope these additions and clarifications answer your concerns directly, and we would appreciate it if you could consider updating your score accordingly.
References:
Andrich & Luo (1993), Appl. Psychol. Meas. 17(3), 253–276.
Johnson & Junker (2003), J. Educ. Behav. Stat. 28(3), 195–230.