- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors explore the use of multidimensional IRT (item response theory) as an evaluation paradigm that is more stable in ranking models and more interpretable. They use a taxonomy created by a human-in-the-loop process where an LLM is prompted to create candidate item level ability categories which are then arranged hierarchically and then the final taxonomy is annotated on benchmark items by a prompted LM as well. These item-ability annotations provide a so-called "Q matrix" which which is used to mask items that are not relevant to a given ability when fitting the IRT model. They adapt the NeuralCD framework (Wang et al., 2023) which essentially uses a 2 parameter IRT set up for each ability category to build a vector x of all the abilities which is input to an MLP and trained against the correct or incorrect item level responses. To assess their approach the authors provide a correlation between estimated ability scores and the model accuracies on the items used to fit the ability estimates, a correlation between abilities on different benchmarks when fitting on each benchmark separately, and one held out evaluation where they predict unseen item responses based solely on the LLM assigned ability categories and the fitted model ability estimates. In this third metric they compare against Adele, another multidimensional IRT baseline and find that their method gets better AUC on predictions. They then conduct the same set of experiments on 3 additional domains but do not compare against Adele on those.
Critical strengths
- The paper engages with a very real problem of model ranking instability across benchmarks, and I believe IRT is a very promising avenue to approach this problem.
- They demonstrate that a straight forward mIRT method can get good correlation across benchmarks within a domain and can predict held out items based on what abilities they are judged as requiring.
- They provide the prompts used for their LM-assisted taxonomy creation process which could help implement this kind of approach over new domains.
Critical weaknesses
- The paper only compares to one baseline (Adele) in one experiment even though many other methods from previous work would be appropriate baselines. It's also not clear why the authors compare to the Adele only in the math domain but not in the other 3 domains (I believe it should be possible since Adele is not domain specific). Relatedly the authors do not describe how exactly they make unseen item predictions with Adele. They simply say they use "an identical procedure" to the one they use on their approach but it's not clear to me how that would work with Adele.
- They make factually wrong claims about related work: at line 459 they describe a list of citations including Rodriguez et al. (2021) as "predominantly model a single latent trait and do not estimate multidimensional ability profiles". But Rodriguez explicitly explores multidimensional IRT with their "IRT-vec". The paper also does not cite other significant mIRT work like earlier work like tinyBenchmarks (Maia Polo et al., 2024). The only mIRT work that they engage with is Adele and they claim (line 070) "it still operates on a fixed, predefined ability taxonomy, treating model abilities as immutable dimensions rather than dynamic, context-dependent constructs jointly shaped by benchmark designs and task contexts." It seems a stretch to say that this distinguishes Adele from their work since the paper's method also creates a fixed taxonomy through a human-in-the-loop taxonomy building process, and nothing prevents the Adele approach from similarly being used with different domain specific taxonomies produced by a manual process.
- The methods in the paper are only partially defined in the body of the paper. For instance section 2.4 which defines the paper's IRT model, just defines how x is built and then says it "is the input to the subsequent MLP layer" but does not describe how that MLP is optimized or what its targets are. This information is only available through inspection of the anonymous code repo and appendix.
Questions for authors
How exactly did you make predictions with Adele for your results in figure 5? And is there a reason you could not include Adele results on your 3 additional domains?
Dear authors,
thanks for putting in the time to reply to my comments!
I am afraid that I won't be able to address my score unless those two points are really addressed
For me, this would be addressed by concrete experimental findings: Concrete measurements of speedups or insights I could not gain with any other method.
To me, the current framework reads like one more other approach to analyze multiple benchmarks in a more cohesive way -- but beyond the others telling me it is better, I do not see clear experimental data showing me that it really is better. (This also shows in the discussion here: Much of the discussion has been around what previous work did - not not discussing concrete experimental findings in this paper)