Sociolectal Analysis of Pretrained Language Models
Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Standard
Sociolectal Analysis of Pretrained Language Models. / Zhang, Sheng ; Zhang, Xin ; Zhang, Weiming ; Søgaard, Anders.
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021. p. 4581–4588.Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - GEN
T1 - Sociolectal Analysis of Pretrained Language Models
AU - Zhang, Sheng
AU - Zhang, Xin
AU - Zhang, Weiming
AU - Søgaard, Anders
PY - 2021
Y1 - 2021
N2 - Using data from English cloze tests, in which subjects also self-reported their gender, age, education, and race, we examine performance differences of pretrained language models across demographic groups, defined by these (protected) attributes. We demonstrate wide performance gaps across demographic groups and show that pretrained language models systematically disfavor young non-white male speakers; i.e., not only do pretrained language models learn social biases (stereotypical associations) – pretrained language models also learn sociolectal biases, learning to speak more like some than like others. We show, however, that, with the exception of BERT models, larger pretrained language models reduce some the performance gaps between majority and minority groups.
AB - Using data from English cloze tests, in which subjects also self-reported their gender, age, education, and race, we examine performance differences of pretrained language models across demographic groups, defined by these (protected) attributes. We demonstrate wide performance gaps across demographic groups and show that pretrained language models systematically disfavor young non-white male speakers; i.e., not only do pretrained language models learn social biases (stereotypical associations) – pretrained language models also learn sociolectal biases, learning to speak more like some than like others. We show, however, that, with the exception of BERT models, larger pretrained language models reduce some the performance gaps between majority and minority groups.
U2 - 10.18653/v1/2021.emnlp-main.375
DO - 10.18653/v1/2021.emnlp-main.375
M3 - Article in proceedings
SP - 4581
EP - 4588
BT - Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
PB - Association for Computational Linguistics
T2 - 2021 Conference on Empirical Methods in Natural Language Processing
Y2 - 7 November 2021 through 11 November 2021
ER -
ID: 299822479