On Cross-Dataset Generalization in Automatic Detection of Online Abuse
release_tr6njwf2nzbvvl4a35an7eaeji
by
Isar Nejadgholi, Svetlana Kiritchenko
2021
Abstract
NLP research has attained high performances in abusive language detection as
a supervised classification task. While in research settings, training and test
datasets are usually obtained from similar data samples, in practice systems
are often applied on data that are different from the training set in topic and
class distributions. Also, the ambiguity in class definitions inherited in this
task aggravates the discrepancies between source and target datasets. We
explore the topic bias and the task formulation bias in cross-dataset
generalization. We show that the benign examples in the Wikipedia Detox dataset
are biased towards platform-specific topics. We identify these examples using
unsupervised topic modeling and manual inspection of topics' keywords. Removing
these topics increases cross-dataset generalization, without reducing in-domain
classification performance. For a robust dataset design, we suggest applying
inexpensive unsupervised methods to inspect the collected data and downsize the
non-generalizable content before manually annotating for class labels.
In text/plain
format
Archived Content
There are no accessible files associated with this release. You could check other releases for this work for an accessible version.
Know of a fulltext copy of on the public web? Submit a URL and we will archive it
2010.07414v3
access all versions, variants, and formats of this works (eg, pre-prints)