- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes an method to automatically generate to evaluate LLMs for relational reasoning even when it is not clear what makes problem instances hard. The method uses Edge Transformers with Datalog rules, performing evolutionary search to identify samplers that can generate challenging problem instances. It is demonstrated that training ET on generated hard examples enhanced their performance.
Critical strengths
- The paper is well-written and easy to follow. It delivers important concepts on relational learning. The idea of automating the evaluation of relational reasoning is interesting and insightful.
- The proposed framework is model-agnostic, and thus should transfer to different architectures. The samplers induced by LLMs successfully identify new forms of difficulty that cannot be explained via standard criteria such as inference depth. This demonstrates the practical efficacy of the proposed method.
Critical weaknesses
- The framework aims at generating challenging tasks, but it only tests the models on small instances. It is noted in experiments that KGs contain at most eight entities. Real-world reasoning tasks typically involve with a large number of data. Thus a concern on the scalability remains.
Questions for authors
- The system generates instances by deliberately search for ones where models get wrong. Would it be ensured that the method generates genuinely hard instances, rather than identifying mere bugs or blind spots for given models?
Our choice to generate small KGs is not a matter of scalability. It is straightforward to generate reasoning problems that are challenging for neural models, when the problems are allowed to be large. For instance, the NoRA 1.1 dataset was created by recursively expanding story graphs using a fixed set of templates. This makes it straightforward to generate problems with arbitrarily large OPEC values, which can reduce the accuracy of all existing models to practically 0. However, when models fail on small problems, it suggests that there is something fundamentally missing in their architecture or training data, which is precisely what we want to uncover. Note that reasoning performance on small KGs is also important for many applications (e.g. relation extraction in NLP, molecular property prediction).
In fact, finding “blind spots” for a given model is one of our main motivations, so we disagree that finding “mere bugs or blind spots” would be a weakness. However, our transferability experiments (e.g. Table 5) show that the learned priority functions transfer to other model architectures, and to some extent also to other rule bases, suggesting that they do capture general difficulty aspects, rather than model-specific artefacts.