Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures
Tyler Chang, Catherine Arnett, Abdelrahman Sadallah, Abdelrahman Eldesokey, Abeer Kashar, Daud Abolade, Abosede G Olanihun, Adam Labaran M, Adhikarimayum Meerajita Sharma, Aditi Gupta, Adril P Merin, Adwoa A Bremang, Afitab Iyigun, Afonso Simplício, Ahmed Essouaied, Aicha Chorana, Akhil Eppa, Akintunde Oladipo, Akriti Kuri, Akshay Ramesh, Aleksei Dorkin, Alfred M Kondoro, Alham Aji, Ali E Çetintaş, Allan Hanbury, Alp Niksarli, Alvaro Arroyo, Amin Bajand, Amol Khanna, Ana Chkhaidze, Ana C Condez, Anamaria Roberta Hartl, Andiswa Mkhonto, Andrew Hoblitzell, Angelos Poulis, Anirban Majumder, Anjali Chaudhary, Anna Vacalopoulou, Annika Simonsen, Anton Kovalev, Anupam Nayak, Ashvanth.S, Ayodeji Lana, Ayu Purwarianti, Bashar Alhafni, C. Benedict C. Busole, Bernard Ghanem, Bharti Nathani, Biljana Stojanovska Đurić, Ogundipe B Funmilola, Bolaotan Agbonile, Bruce Torres Fischer, Burak Tutar, Burcu Çınar, Cade Kane, Can Udomcharoenchaikit, Chadi Helwe, Cecilia Liu, Chiamaka G Nwokolo, Christopher Homan, Clément Sampebgo, Cristina España-Bonet, Cynthia Amol, Daeyeop Lee, Dan Saattrup Smart, Dana Arad, Daniil Dzenhaliou, Dasol Choi, David Liu, David Semedo, David Anugraha, OLAOYE D Oluwaseun, Deborah Popoola, Deividas Mataciunas, Dennis Asamoah Owusu, Dhyuthy Krishna Kumar, Diogo Tavares, Diogo Glória-Silva, Divyanshu Goyal, DongGeon Lee, Estefany Kelly Buchanan, Ebele N Anajemba, Ngozi G Egonu, Elias Herranen, Eman Nisar, Emile Anand, Emuobonuvie M Ajiboye, Eryawan P Yulianrifat, Esther Adenuga, Ewa Rudnicka, Faith O Itiola, Fareeha Fayyaz Sheikh, Fathima Thekkekara, Fatima Haouari, Faustin NSENGIYUMVA, Fenal A Ilasariya, Filbert A Tjiaranata, Firas Laakom, Francesca Grasso, Francesco Periti, Francesco Orabona, Gbenga K Solomon, Genta Winata, Nghia G Ngo, Gloria Udhedhe-oze, Gonçalo Vinagre, Gopi N Challagolla, Gorka Urbizu-Garmendia, Gouthami Vadithya, Guijin Son, Gyan S Mohapatra, Hafeez Ullah, Hafsteinn Einarsson, Hai Hu, Hamidreza Saffari, Hamza Zaidi, Haopeng Zhang, Harethah Abu Shairah, Hele-Andra Kuulmets, Hitesh Patel, Houda Bouamor, Hwanjo Yu, Iben N Debess, İbrahim Ethem Deveci, Ikhlasul A Hanif, Ikhyun Cho, Inês Vieira, Inês Calvo, Ismail O Daud, Yusuf I Abayomi, Itay Itzhak, Ivan Zhelyazkov, Ivan Belashkin, Ivan Spada, Jacob Brinton, Jafar Isbarov, Jaka Čibej, Jan Kocon, Jan Cuhel, Jauza A Krito, Jebish Purbey, Jennifer Za, Jennifer Mickel, Jenny Kunz, Jessica Ratovondranto, Jeyarajalingam Varsha, Jihae Jeong, Jimena T Dávalos, Jinu Lee, Joao Magalhaes, John S Yi, Jongin Kim, Joseph Chataignon, Joseph Marvin Imperial, Jubeerathan Thevakumar, Judith Land, Iuliia Alekseenko, Jiang Junchen, Jungwhan Kim, Kairit Sirts, Kamesh R, Kanda Patrick Tshinu, Kätriin Kukk, Kaustubh Ponkshe, Kavsar Huseynova, Ke He, Kenneth Enevoldsen, Kent J Alvarez, Kerem Zaman, Khalil Mrini, Kian Kyars, Komal Gour, Krishnakumar Lainitha, Krister Kruusmaa, Kunal Mukherjee, Kusum Chouhan, Laura Castro, Laura M Porrino-Moscoso, Lenny S Nzambi, Leshem Choshen, Lilja Øvrelid, Lisa Alazraki, Lovina Ehimen-Ugbede, Luheerathan Thevakumar, Luxshan Thavarasa, Mamadou K KEITA, Mansi Jangid, Marco De Santis, Marcos Garcia, Marek Suppa, Mariam D'Ciofalo, Marii Ojastu, Marium Atta, Maryam Sikander, Maximos Skandalis, Mehak Soomro, Mehmet İlteriş Bozkurt, Melaku Bayu, Menan Velayuthan, Michael Leventhal
Abstract
To date, there exist almost no culturally-specific evaluation benchmarks for large language models (LLMs) that cover a large number of languages and cultures. In this paper, we present Global PIQA, a participatory commonsense reasoning benchmark for over 100 languages, constructed by hand by over 350 researchers from over 65 countries around the world. The 141 language varieties in Global PIQA cover five continents, 19 language families, and 24 writing systems. In the non-parallel split of Global PIQA, over 50% of examples reference local foods, customs, traditions, or other culturally-specific elements. In the parallel split, we translate more "culturally agnostic" commonsense reasoning questions into 131 language varieties, for direct cross-lingual comparisons. In both splits, all examples have been verified by native speakers of the languages. We find that state-of-the-art LLMs perform well on Global PIQA in aggregate, but they exhibit weaker performance in lower-resource languages (e.g. up to a 68% accuracy gap between languages in the parallel split). Global PIQA highlights that in many languages and cultures, everyday knowledge remains an area for improvement in LLMs, alongside more widely-discussed capabilities such as complex reasoning and expert knowledge. Beyond its uses for LLM evaluation, Global PIQA provides a glimpse into the wide diversity of cultures in which human language is embedded.