Announcement 23 11 06

2023

1. How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweets

   Danula Hettiachchi, Indigo Holcombe-James, Stephanie Livingstone, Anjalee Silva, Matthew Lease, Flora D Salim, and Mark Sanderson

   In Proceedings of the Eleventh AAAI Conference on Human Computation and Crowdsourcing (HCOMP ’23), 2023

   Abs Bib PDF DOI

   Crowdsourced annotation is vital to both collecting labelled data to train and test automated content moderation systems and to support human-in-the-loop review of system decisions. However, annotation tasks such as judging hate speech are subjective and thus highly sensitive to biases stemming from annotator beliefs, characteristics and demographics. We conduct two crowdsourcing studies on Mechanical Turk to examine annotator bias in labelling sexist and misogynistic hate speech. Results from 109 annotators show that annotator political inclination, moral integrity, personality traits, and sexist attitudes significantly impact annotation accuracy and the tendency to tag content as hate speech. In addition, semi-structured interviews with nine crowd workers provide further insights regarding the influence of subjectivity on annotations. In exploring how workers interpret a task - shaped by complex negotiations between platform structures, task instructions, subjective motivations, and external contextual factors - we see annotations not only impacted by worker factors but also simultaneously shaped by the structures under which they labour.

   @inproceedings{Hettiachchi2023-hj,
     title = {How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweets},
     booktitle = {Proceedings of the Eleventh {AAAI} Conference on Human Computation and
                    Crowdsourcing},
     author = {Hettiachchi, Danula and Holcombe-James, Indigo and Livingstone, Stephanie and de Silva, Anjalee and Lease, Matthew and Salim, Flora D and Sanderson, Mark},
     series = {HCOMP '23},
     volume = {11},
     number = {1},
     year = {2023},
     publisher = {AAAI},
     doi = {10.1609/hcomp.v11i1.27546},
     dimensions = {true}
   }