- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RedTeachingBench, a benchmark for evaluating how small language models respond to false premises in teacher requests for educational flashcards. It contains 500 curriculum items spanning six subjects in French lower-secondary education. The main evaluation covers 22 open-weight models and compares their behavior with AdvBench and FalseQA.
Critical strengths
The paper focuses on teachers preparing instructional material, where an apparently legitimate request can contain a factual mistake. Grounding the evaluation in a specific curriculum provides a concrete use case and connects benchmark content to knowledge students are expected to acquire.
Critical weaknesses
The comparisons do not isolate the contribution of curriculum grounding. FalseQA uses a plain question-answering instruction, whereas RedTeachingBench adds a teacher persona, a flashcard task, and an instruction to preserve the exact question. Differences therefore conflate content with prompting and output-format requirements. The authors acknowledge that cross-benchmark metrics differ, but these comparisons still underpin the claimed distinct robustness property.
Judge reliability is insufficiently established. The validation primarily consists of agreement between two model judges and manual inspection of 50 disagreement cases. This does not measure accuracy on a representative sample or detect errors shared by both judges. In addition, the judges are themselves among the evaluated models.
The primary metric is not well aligned with the stated educational objective. Section 4.2 treats answers that preserve the question but correct its premise as non-failures, yet the prominent 86.7% versus 7.9% comparison excludes these answers from RedTeachingBench resistance. Although the broader figure of approximately 41% is also reported, the emphasis on 7.9% can overstate the failure to identify erroneous premises.
Presentation quality need inspection. Figure 1 is extremely small despite substantial surrounding whitespace, and labels in Figures 2 and 3 are difficult to read. Table 7 extends beyond the main text block. These issues should be corrected to make the manuscript readable.
Questions for authors
Could you evaluate both FalseQA and RedTeachingBench under matched prompting conditions, including plain question answering and the same teacher/flashcard wrapper?
How do models perform on the corresponding true-premise questions? Could you report over-rejection rates and false-premise performance restricted to items whose underlying knowledge each model demonstrates on the uncorrupted version?