arrowCIRSS Home arrow Publications arrow Publication Detail

Evaluating Rater Quality and Rating Difficulty in Online Annotation Activities

Full APA Reference

Organisciak, P., Efron, M., Fenlon, K., & Senseney, M. (2012, October). Evaluating Rater Quality and Rating Difficulty in Online Annotation Activities. Proceedings of the Annual Meeting of the American Society for Information Science and Technology, 49.

Publication Abstract

Gathering annotations from non-expert online raters is an attractive method for quickly completing large-scale annotation tasks, but the increased possibility of unreliable annotators and diminished work quality remains a cause for concern. In the context of information retrieval, where human-encoded relevance judgments underlie the evaluation of new systems and methods, the ability to quickly and reliably collect trustworthy annotations allows for quicker development and iteration of research. This study evaluates indicators of non-expert performance along three lines: temporality, experience, and agreement. In doing so, it finds that early user performance is a key indicator for future performance. Additionally, the time spent by raters familiarizing themselves with a new set of tasks is important for rater quality, as is long-term familiarity with a topic being rated. These findings may inform large-scale digital collections' use of non-expert raters for performing more purposive and affordable online annotation activities.