- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Summary
The paper presents an investigation of a new curriculum learning approach. The approach entails the "bucketing" of data points into buckets of increasing difficulty. Different to other methods, they define this difficulty by the "capability index" of the least capable model that can correctly predict them. They have a reasonable definition of this capability index congruent with current understanding (it basically entails looking at model size or number of few shot examples).
Recommendation
It's not stellar and there is on experimental limitation that might cause me to revise my score downwards or upwards (see below), but this is an interesting investigation on a very interesting topic with a fresh take and a very clear story. The language is clear and the paper is easy to follow. The authors have left themselves very little space to interpret their findings (discussion is only 1 paragraph long), which I think is a major downside of this paper (see below).
At this stage, I will give this paper a "weak accept" because I have some misgivings about some aspects, but all in all I think it deserves publication and is a nice addition to the literature on curriculum learning
(Even if the paper is not accepted now and even if the issue I identify with evaluation is indeed true, I still recommend the authors revise and try to publish elsewhere because the idea is truly interesting and I think the community needs to see the results)
Reviewer self-evaluation
I have been working on curriculum learning on-and-off for the last 5-6 years, primarily on domain-specific aspects of it relevant to my own field. My field is speech (primarily audio, less so text). I am thus very familiar with NLP methodologies and curriculum learning, though I haven't done much work on the intersection of the two.
Critical strengths
New definition of difficulty
The authors give a new and very interesting definition of instance difficulty (the score that is used to rank examples during curriculum learning)
They define it based on model competence, evaluating three definitions of competence:
- Model size
- Checkpointing (i.e. training for longer/shorter)
- Length of the context window
They then consistently show that
- Easy-to-hard curricula using their metrics outperform IID training and relevant curriculum learning baselines
- Hard-to-easy curricula underperform IID (though not all other easy-to-hard baselines in all cases)
They do so on two domains (math and chess)
This is a very interesting finding from both a practical and a theoretical perspective:
- Practical perspective: It helps train smaller LMs better (also faster? but this doesn't seem to have been checked..)
- Theoretical perspective: I think this has interesting repercussions for understanding these models and problems. Alas, this is where I think that the authors erred by investing too little time in interpretation (see below)
A clear story
I said this before but I want to re-emphasize it (I guess I'm biased because I just reviewed too many papers who tried to do too many things at once) -- the authors have a clear story from beginning to end. All experiments serve that story. There are no random or irrelevant side-quests. The reader is guided into an investigation and leaves it with more information than they came in. In my case, as someone interested in curriculum learning, this also left me with more ideas for my own research and more questions that I would like to ask the authors during the conference (I probably won't be there, I hope that the authors ultimately make it).
In short, I believe this is precisely the type of paper that should be published at ICLR..
Alas, I have some misgivings which should be addressed first...
Critical weaknesses
No space for discussion
I understand that the page limitation is too limiting (everyone seems to be struggling with this), but it's there for some reason and authors are supposed to take it or leave it (I also struggle with this)
So on the one hand, the authors have to fill in a lot that deserves to be there, but on the other hand, they're left without any space to interpret their findings.
However this is precisely the thing we readers are waiting for! The authors do have plenty of interesting observations. Do they really need Figure 1? Can they not cut some parts of the introduction or related work, which is indeed thorough and relevant but doesn't leave any space for the more interesting part?
Take Figure 10 for example. The authors note that LevelUp (easy-to-hard) starts with a much larger gradient than IID and gradually decays towards it, while LevelDown (hard-to-easy) starts close and deviates later.
The authors then go on to note that "[...], indicating that an effective curriculum exposes the learner to a qualitatively different optimization problem."
Different how??!?!?!?!? This is precisely what the reader wants to find out. Why do "easy" examples have a higher gradient norm? And why do most learning rate recipes for transformers "warm up" the learning rate, thus starting from softer updates and gradually ramping up, as LevelDown appears to be doing. Is there any connections there?
There is so much that can be unpacked from these observations and it is a pity that the authors leave themselves with just enough space to say that "we tried it and it worked". I strongly suggest them to find a way to get more interpretation into their work (a very hard task, I know!)
Figure 6 vs Human Curricula
Figure 6 is the only attempt to give some depth to the interpretation.
There seems to be a monotonic relationship between "Average Question Length" and "Levels" (although the figure is missing error bars!)
So, on first interpretation, those two curricula should be interchangeable if the same pacing rule is applied. Yet LevelUp consistently outperforms the human curricula, making me wonder if the pacing was exactly the same.
Specifically, given that you essentially create hard "buckets" using the transitional problem definition, which gives you a discrete difficulty value, how do you handle curricula with "softer" values, like the question length? Because if you apply a split based on Figure 6 (leftmost), then you should get identical data buckets as for LevelUp, right? Then you should also get the same performance..Unless the error bars are too high and then the ordering is indeed different..
Questions for authors
Like I said in the introduction, I would appreciate an answer to my question regarding Figure 6 and Human Curricula. This could cause me to change my score both upwards and downwards
Beyond that, I don't have a concrete question, but kindly ask the authors to consider the broader repercussions of their findings, and perhaps to state them here (and/or revise their discussion accordingly). What are we to make of this?
3 comments
Not convinced, keeping my score
Thanks for taking the time to respond, but I'm not (yet?) convinced to change my score.
-
I only see a very loose connection with ZPD. In fact, it's so loose that it can be applied to all curriculum learning methods, thus rendering it almost vacuous. You are using the difficulty scores of progressively more capable models to identify easy-to-hard rankings of the training data. This does not say anything about whether harder samples are outside of the ZPD for the current model. (It also doesn't say anything about how this should be measured.) For instance, you do not evaluate whether the model agrees with the ranking of the other models or so.
-
Figure 6 vs human curricula: Okay, the boxplots appear to have a wider variability than I expected. Isn't that the main takeaway? So doesn't that invalidate what you were trying to say there?
Anyways, these answers don't convince me to increase my score further, but the also don't cause me any additional concerns, so I'll keep it the same.
Clarification on ZPD connection + Figure 6
Thank you for your follow-up questions. We greatly appreciate your continued engagement and the opportunity to discuss our work in detail. Below we further clarify our stance on these two aspects of our work, and hope you will consider raising your score.
1. Connection to the ZPD
- We hypothesize that transitional problems represent the ZPD because they are the problems that distinguish one competence level from another — they are the problems that are just out of reach of a model at one level but crucially, solved by the next levels of models.
- In contrast, static domain-specific curricula do not explicitly order difficulty in a way that's specific to the learner. Dynamic curricula capture some notion of the ZPD, but aside from being expensive to run, they are short sighted, and thus what is out of reach may not actually represent the ideal direction in which development (i.e., improvement towards a stronger model) should occur.
- Our problem ranking relies on the full series, and changing the order of the series will change the ranking. Since we order by performance on the target task, transitional problems represent a progression in competence relative to the learner (weakest model), following the idea of scaffolding/the ZPD.
2. Two key takeaways from Figure 6
- Our main goal in showing Figure 6 is that our procedure for selecting transitional problems on a task, which is derived entirely from the performance of weak and strong models on that task, is still weakly aligned with some measures of human difficulty. This is neither a perfect recreation (which would be shown by a very narrow boxplot) nor is it a complete lack of alignment (which would be shown by an uncorrelated mean).
- Thus, the trends in Figure 6 and the boxplots can be interpreted in two ways: as a 'sanity check' that higher levels of transitional problems do align with our intuition of what is hard, but also as a way to show that they can't be reconstructed from a few such domain-specific measures alone. The boxplots demonstrate both of these points, and we thank the reviewer for highlighting this result.
Response to Reviewer 2tec
We thank the reviewer for their thoughtful and detailed review on our work. We are grateful to the reviewer for accurately capturing the spirit of our work, and for their positive comments and acceptance recommendation. We agree that our results merit additional discussion and present an extended analysis of our results below, with an emphasis on the reviewer’s primary concern: our interpretability figures. We hope that our direct and detailed response will positively influence the reviewer’s recommendation of our work.
Extended Discussion
Overall, our objective in this work is to adapt multiple successful ideas from human curricula — the tiered education system and the Zone of Proximal Development (ZPD) — into a machine learning system. As we outline in section 1, human curricula are successful in part because they are calibrated to the individual needs of weaker and stronger learners, and because they emphasize what is just out of reach. By defining the transitional training problems that characterize this specific notion of difficulty for a learner, we are able to improve upon existing curriculum strategies even with a simple, linearly structured training regime.
Analysis of Figure 6 vs. human curricula
On the broader repercussions of our work
We present a novel view on the connection between curricula designed by human teachers and those for ML models. Even within the same syllabus, teachers/tutors often calibrate their exact curricula to what a weak or strong student may learn. Our method is in a similar spirit, as it modulates the data diet to a model based on what it performs poorly on, but is still achievable by other models. This is an interpretation of the Zone of Proximal Development (ZPD) concept in psychology that we describe in Appendix A. We will add this and the above discussion to the final version of our paper, and we thank the reviewer for sparking this discussion which adds value to our findings.