A user study on people’s perception to the credibility of online health information

Judging the credibility of information is a subjective process and prone to biases. This issue can be especially concerning in health information seeking. Some efforts have been made to define robust credibility assessment guidelines that support the development of reliable test collections. This is of the utmost importance since the applicability of retrieval algorithms to real use case scenarios relies on the quality of the labelled data. Yet, the question persists as to whether the labels created by these guidelines can effectively serve as a surrogate for the genuine judgements of credibility as perceived by end-users. Motivated by this, we conducted a user study with 1,000 participants. We demonstrate that there is a correlation between participants’ judgements and the reference values produced following existing guidelines. Further analyses of the data reveal worrying insights into people’s ability to judge the credibility of online medical content, leading to potential personal harm.

keywords: Health-related content, Credibility, User study