The Difficulty of Misinformation Labelling: A Case Study for Radon Gas-Related Searches

The creation of labelled collections related to misinformation is a crucial aspect in the development of automatic technologies that filter harmful content. This is particularly important for health-related risks. However, assigning quality labels (e.g., correctness or credibility) to texts needs to be done rigorously. In this article, we describe our endeavours to build a collection of labelled passages, with relevance and quality annotations, for search tasks related to the risks of radon gas. In addition to illustrating the difficulties encountered in a labelling project of this kind, our contribution with this work is to provide the scientific community with a new annotated resource that can be used in the future to support supervised learning in this area.

keywords: Web Search, Misinformation, Radon Gas, Labelling